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CHAPTER 1: THE CONTEXT OF ACADEMIC 


ACHIEVEMENT AND STEM EDUCATION 


“The most important thing we can do is inspire young minds and to advance the kind of 
science, math, and technology education that will help youngsters take us to the next 
phase of space travel.” 


John Glenn (2000) 


On July 20, 1969, three American astronauts, Neil Armstrong, Buzz Aldrin, and Mike Collins 
completed the first manned mission to the moon. Over 650 million people watched on 
television in eager anticipation, the most viewed event in history at the time, as Armstrong 
took the first human steps on Earth’s only natural satellite (National Aeronautics and Space 
Administration [NASA], 2020). Fueled by the Space Race with the Soviet Union beginning 
in the mid-1950s, the United States was able to relinquish technological superiority with a 
few human steps (Space Race, 2010). This pivotal event sparked American interest in 
science, technology, engineering, and mathematics which has since been promoted many 


times after this “giant leap for mankind” in American history (NASA, 2020). 


The Space Race, commencing in the 1950s, produced many science initiatives under 
President Dwight D. Eisenhower due to the increasing national belief that scientists in the 
Soviet Union were surpassing U.S. scientists in advancing technologies. Aside from NASA, 
the National Defense Education Act (NDEA) signed in 1958 demonstrated the federal 
government’s continuing involvement in public education and gave funding to both public 
schools and college institutions. The two primary goals of the NDEA were to produce 
military and college graduates with advanced technological skills and provide financial 
support to college students through the National Defense Student Loan (NDSL) program 
(Schwegler, 1982). That same decade, the Defense Advanced Research Projects Agency 
(DARPA) was created with the sole mission still stated today: “To make pivotal investments 


in breakthrough technologies for national security” (DARPA Editors, 2020). 


Ht 


The Context of Academic Achievement and STEM Education 


In 1983, A Nation at Risk was published as an educational reform policy reviving the 
importance of science, technology, engineering, and mathematics in public education. This 
landmark report in American educational history unveiled the common misconception among 
the public that Americans were leading the world in production, commerce, and education. 
The report shared some staggering statistics: 23 million American adults were functionally 
illiterate and the average achievement of high school students on standardized tests was lower 
than it was 26 years prior when Sputnik was launched (National Commission for Excellence 
in Education [NCEE], 1983). The latter was a particularly hard pill to swallow for Americans 
who remembered watching the Moon landing with national pride. A Nation at Risk reflected 
upon the dismal state of the American educational system. Shocked Americans were to learn 
that not only did they not lead the free world in education and advancing technologies, rather 


they lagged behind: 


The time is long past when American's [sic] destiny was assured simply by an abundance of 
natural resources and inexhaustible human enthusiasm, and by our relative isolation from the 
malignant problems of older civilizations. The world is indeed one global village. We live 
among determined, well-educated, and strongly motivated competitors. We compete with 
them for international standing and markets, not only with products but also with the ideas of 
our laboratories and neighborhood workshops. America's position in the world may once 


have been reasonably secure... it is no longer. (NCEE, 1983) 


The risk is not only that the Japanese make automobiles more efficiently than Americans and 
have government subsidies for development and export. It is not just that the South Koreans 
recently built the world's most efficient steel mill, or that American machine tools, once the 
pride of the world, are being displaced by German products. It is also that these developments 
signify a redistribution of trained capability throughout the globe. Knowledge, learning, 
information, and skilled intelligence are the new raw materials of international commerce and 
are today spreading throughout the world as vigorously as miracle drugs, synthetic fertilizers, 
and blue jeans did earlier. If only to keep and improve on the slim competitive edge we still 
retain in world markets, we must dedicate ourselves to the reform of our educational system 
for the benefit of all - old and young alike, affluent, and poor, majority and minority. 
Learning is the indispensable investment required for success in the ‘information age’ we are 


entering (NCEE, 1983, p. 10). 


2 


A Pathway to Success? A Longitudinal Study Using Hierarchical Linear Modeling of Student and 
School Effects on Academic Achievement in a Middle School STEM Program 


The echoes of these words from almost 40 years ago were particularly poignant as the nation 
faced economic turmoil, flailing infrastructure, and a need for advanced medical technologies 
in the wake of the Covid-19 pandemic that struck the nation in early 2020. The need for 
science, technology, education, and mathematics in education is becoming more and more 
important as the nation works in collaboration with business, medicine, and commerce to 
develop cures and vaccines to eradicate a virus plaguing modern society. Our nation is at risk 


now, just as it was in 1983. 


The 1980s was a monumental time in calling to light the development of STEM education 
particularly through the American Association for the Advancement of Science (AAAS). The 
AAAS was founded in 1848 with the goal of increasing collaboration and exchanging of 
information among scientists to advance scientific progress at an increasing pace. Currently, 
it is the largest scientific organization in the world with over 120,000 members comprising 
experts in astronomy, education, engineering, medical sciences, etc. In 1985, sparked by the 
passing of Haley’s Comet, the AAAS created Project 2061. This long-term scientific 
initiative aimed at improving American technological, mathematic, and scientific literacy. 
Project 2061 developed a comprehensive set of K-12 learning goals with state and national 
science standards creating the Next Generation Science Standards. In addition, the AAAS 
formed the Atlas of Science Literacy, Benchmarks for Science Literacy, and Science for All 
Americans. The latest defines science literacy and specifically explains standards and 
curriculum topics, best practices in learning and teaching, and basics of educational reform 


(AAAS Editors, 2020). 


In 2012, recognizing the need for further STEM education policy, President Barack Obama 
created the Educate to Innovate campaign promising to prepare the nation’s youth in the 
areas of STEM education, computer science, and technological literacy. During President 
Obama’s first term, he emphasized the importance of math and science preparation through 
funding under the Elementary and Secondary Education Act (ESEA), Race to the Top (Rttt), 
and the Investing in Innovation Fund (i3). During Obama’s second term, he shocked the 
nation by proclaiming during his 2013 state of the union address the coming of a “Sputnik 
moment” with the facilitation of government investment in infrastructure, education, and 
research (Obama, 2013). In addition, three billion dollars was given to 14 federal agencies 
such as the U.S. Department of Education, National Science Foundation (NSF), and the 


Department of Energy to invest in STEM education programs to provide STEM education 
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and computer science programming (Handelson & Smith, 2016). President Obama 
emphasized, during his STEM for All speech at the National Science Fair in 2013, the 
importance of an “all-hands-on-deck approach to science, technology, engineering, and 
math... We need... to make sure that all of us as a country are lifting up these subjects for the 


respect they deserve” (Obama, 2013). 


Shortly after the Education to Innovate campaign, President Obama signed the STEM 
Education Act of 2015 expanding training and research programs to teachers in math and 
science. This occurred via the National Science Foundation (NSF) scholarship program, 
improving informal STEM education research, and formally adding computer science into the 


definition of STEM education (STEM Education Act, 2015). 


That same year, and several decades after A Nation at Risk was released, the Every Student 
Succeeds Act (ESSA) was signed in 2015. This educational reform and policy replaced the 
No Child Left Behind (NCLB) legislation from 2001 and transferred more power to local 
entities and states (Achieve, 2017). The passage of ESSA has given local school districts the 
decision-making power and funding to provide formal STEM programming, digital literacy 


curriculum, and the purchasing of technology (ESSA, 2017). 


In 2019, two bills were passed through Congress further enhancing public education’s ability 
to provide STEM programming to meet the needs of all students including underrepresented 
populations. The STEM Research and Education Effectiveness and Transparency Act was 
passed allowing for broadening participation in NSF research and educational programs 
including STEM education (STEM Research and Education Effectiveness and Transparency 
Act, 2019). The second bill, the 2/st Century STEM for Girls and Underrepresented 
Minorities Act provided grants to local educational agencies to empower girls and other 
underrepresented minorities to participate in STEM programs and pursue STEM field careers 


(21st Century STEM for Girls and Underrepresented Minorities Act, 2019). 


Throughout the history of STEM education and initiatives, there have been gaps in both 
student participation and academic achievement among underrepresented populations, such 
as students who are economically disadvantaged, minorities, and female students (Beede et 
al., 2011; Gonzalez & Kuenzi, 2012; Hebebci, 2019; Kennedy & Odell, 2014; Sanders, 2009; 
National Science Board, 2019). According to Gonzalez and Kuenzi (2012), many other 


et 
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researchers and economic experts agree that underrepresented groups are an untapped 
resource for potentially filling many STEM-related positions currently needed in the 
workforce. They explained that the exposure to STEM and increase in participation in STEM 
fields among women and racial minorities may pose a solution to the labor shortage described 
by Kennedy and Odell (2014) and Gonzalez et al. (2012). One of the main purposes for the 
formation of the NDEA is to recruit and support underrepresented populations by cultivating 
American talent to join STEM fields. Under NDEA, the federal government developed 
national programs to urge and support women and minorities to pursue careers and higher 


education in scientific and engineering fields (Association of American Universities, 2006). 


All of these government regulated policies discussed such as the NDEA, and later, The STEM 
Research and Education Effectiveness and Transparency Act and the 21st Century STEM for 
Girls and Underrepresented Minorities Act have provided funding for STEM programming 
and curriculums in one way or another. In fact, in 2019, the Department of Education under 
President Trump announced an investment of $540 million to support STEM education via 
research and educational grants. The importance of providing grants for STEM education is 
to propel the mission of the Department of Education. Secretary of Education, Betsy Devos 
stated, “[The mission is] to promote student achievement and preparation for global 
competitiveness by fostering educational excellence and ensuring equal access” (U.S. 


Department of Education, 2018, p. 9096). 


There has been overall minimal research and evaluation on the effectiveness of STEM 
education programs. The last major evaluative report occurred in 2007 with a review by the 
Academic Competitiveness Council ([ACC], 2007) on the effectiveness of federally-funded 
STEM programs. The ACC evaluated 115 STEM programs to assess their effectiveness on 
student achievement and performance. The ACC determined that despite substantial 
investment, there was little evidence of academic benefit, educational gain, or increase in 


students’ performance as a function of these programs (ACC, 2007). 


As the substantial amount of funding for these programs continues to increase, there is a 
significant need to regulate and monitor STEM education and programming. Private sector 
companies are also cashing in on this available revenue source. These companies individually 
develop their own STEM curriculum and programming. Currently, the most popular STEM 


and engineering curriculum companies in the United States are Project Lead the Way 
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(PLTW), Engineering by Design (EbD), EverFi, and STEM Education Works. PLTW is 
presently the curriculum most used nationally (VentureRadar, 2020). Several states are 
providing funding for PLTW programming in their public schools such as Wisconsin, 
Illinois, and Iowa (Schmid, 2007). The direct transfer of taxpayer’s dollars into the pockets of 
private curriculum companies has become a recent issue in STEM education and raises the 
question: How effective is STEM programming on student achievement? Many educational 
leaders and stakeholders wonder if these costly STEM curriculum programs are providing a 
21*-century education and teaching students the necessary skills to be successful both on 


state assessments and in the workforce. 
Statement of the Problem 


Throughout the past two decades, there has been an increase in STEM education initiatives in 
kindergarten through 12" grade and beyond. In addition, funding to public schools to provide 
integrated STEM education from federal and other private sources has increased. There is 
also overall increased need for STEM education programs and instruction. Despite these 
facts, little research has been conducted on the impact of these programs on student 
achievement, particularly in the middle school grades. The effect of STEM programs on 
student success both in school and in the future workforce is an under-researched area in 
education. Additionally, there is ambiguity in defining integrated STEM education among 
educators and researchers. The backbone of STEM education is found in the foundations of 
project-based learning (PBL) and the constructivists education theories of Dewey and Piaget. 
These basic theories assist with defining and explaining the theoretical framework of STEM 
education as it exists today. The broad definitions and effect of STEM programming and PBL 
on student achievement is a necessary topic of research to determine the effectiveness of 


these increasingly popular and often expensive instructional programs. 


Theoretical Framework 


STEM Education 


There are a few fundamental theories supporting STEM education which are pertinent to the 
theoretical framework of this research study. The central theory grounding this research is 
project-based learning PBL theory. Constructivism is another foundational theory supporting 


the work in this study. Other secondary theories framing this study are teacher collective 
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efficacy and student self-efficacy. Both of these concepts account for the teachers’ and 


individual student roles and responsibilities for their own learning. 
Project-Based Learning (PBL) 


The majority of STEM curriculum and programs use PBL theory as the main approach to 
providing STEM instruction. The use of the PBL approach as a method of utilizing student- 
specific pedagogy was first introduced in medical school by Barrows and Tamblyn at 
McMaster University in the 1960s (Hildebrand, 2018). Barrows and Tamblyn felt the 
traditional method of providing medical school education required students to memorize and 
acquire a substantial amount of information that was taught in isolation with no relevance or 
application. This was particularly the case in the first two years of medical school when 
students traditionally did not see patients or have any connections to clinical-based medicine. 
Barrows and Tamblyn developed a PBL curriculum within the four years of the medical 
school program that fostered relevance and application using real-world application and 
interdisciplinary methods (Hildebrand, 2018). Today PBL is used not only in medical schools 
but kindergarten through the 12" grade and in other areas such as business, engineering, law, 


and social sciences (Haggbloom et al., 2002). 


The PBL environment proposed by Tamblyn and Barrows in the 1960s is very similar to the 
method of implementation today, which consists of five characteristics. These, as described 
by Marra et al. (2014), are defined as being problem-focused, student-centered, self-directed, 
self-reflective, and facilitative. All of these fundamental characteristics find their roots in the 


foundations of constructivism theory (Marra et al.). 
Constructivism in Education 


Constructivism is an educational learning theory which relies upon the learner’s 
understanding and prior knowledge of experiences to increase new learning. It is often 
referenced in the philosophies within ontology, ethics, politics, and particularly epistemology 
(Matthews, 1998). In constructivism learning theory, all learning is built upon prior learning, 
or “constructed” from prior knowledge (Nola & Irzik, 2006). The roots of constructivism are 
grounded in educational psychology by Jean Piaget and his development of the theory of 


cognitive development. Piaget’s learning theory concentrated on human development and 
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how humans connected their past experiences and ideas transforming this acquired 
knowledge into new learning (Piaget, 1971). Other educational theorists have expanded on 
Piaget’s work adding their own specialization. Lev Vygotsky’s theory of social 
constructivism described how interactions with adults, other peers, and educational tools are 
used to form new mental constructs within a student’s zone of proximal development 
(Constructivism, 2020). Jerome Brunner stretched Vygotsky’s social constructivism theory 
even further by introducing the idea of instructional scaffolding, or the providing of supports 
which are slowly withdrawn as the learner gains more understanding and ability (Seiffert & 
Sutton, 2009). Several other philosophers and writers have contributed to Piaget’s 
constructivist learning theory, such as Maria Montessori, Wladyslaw Strzeminski, Heinz von 
Foerster, George Kelly, Herbert Simon, Paul Watzlawick, Ernst von Glasersfeld, Edgar 
Morin, and Humberto Maturana (Constructivism, 2020). These scholars, along with many 
others, paved the way for PBL methodologies and the educational pedagogy used in STEM 


education today. 


Most STEM education curriculums and programs like PLTW, EverFi, and PBL programs 
like the All American Soap use the five aspects of the learning cycle developed by Bybee 
(1997) as a foundational learning model derived from constructivist theory. The five aspects 
of the learning cycle theory are: engagement, exploration, explanation, elaboration, and 
evaluation. Engagement involves posing the problem to students, ensuring they make 
connections to previous learning, and pre-assessing existing knowledge. During the 
exploration phase, students actively collect data. Explanation occurs when students apply 
their collected data to the research problem, make connections with their learning, and bring 
in new knowledge and vocabulary words. The fourth phase, elaboration, involves students 
presenting and actively applying their new learning to the research problem. This occurs in 
conjunction as the teachers present new challenges which require critical thinking to solve. 
Lastly, the evaluation phase assesses student’s learning, not necessarily in a pencil-paper test, 
but through presentation of the research problem and solution and final outcomes of the 
project (Bybee, 1997). This five-step learning cycle uses Piaget’s theory of cognitive 
development and requires students to apply prior knowledge and construct new knowledge 


from their learning experiences (Seifert & Sutton, 2009). 


Constructivism is often referenced when discussing pedagogic concepts that encourage active 


learning, or what educational theorist and American philosopher John Dewey identified as, 
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“learning by doing” (Savery, 2006, p. 16). Dewey believed that new learning occurs by 
acquiring new experiences and incorporating it into prior knowledge. His hands-on approach 
to learning paved the way for the foundations of PBL and other progressive educational 
practices used today (Savery, 2006). In My Pedagogic Creed, Dewey stated, "I believe that 
the school must represent present life — life as real and vital to the child as that which he 
carries on in the home, in the neighborhood, or on the playground” (1897, p. 77). Educational 
theories using common sense and straightforward application of acquired knowledge in the 
learning of new skills, such as Dewey’s, is pertinent to the theoretical framework of this 


research study. 
Dewey on Education and Teachers 


The foundational theories of education established by Dewey are most prominent in the 
following works: My Pedagogic Creed (1897), The School and Society (1900), The Child and 
the Curriculum (1902), Democracy and Education (1916), and Experience and Education 
(1938). The latter and his most prominent writings pertain to the theory and application of 
PBL and STEM education (Savery, 2006). Dewey emphasized that learning is a social and 
interactive process with school reform necessary to allow for the gaining of experience and 
interaction with curriculum. He believed that students should take ownership of their learning 
and all students should have an opportunity to learn through experience, particularly through 


“hands-on” learning (Dewey, 1897, p. 78). 


Although Dewey was a prominent believer that students should take responsibility in their 
learning, he also believed the role of the teacher is equally important. Dewey concluded that a 
teacher’s love and passion for learning is more influential to student’s learning than content 
knowledge. According to Dewey, “the best indicator of teacher quality is the ability to watch 
and respond to movement of the mind with keen awareness... as students respond to subject- 
matter presented” (Dewey, 1904, p. 15). Teachers are important when it comes to student 
success in gaining knowledge from a constructivist perspective and in the theoretical 


framework of this research study. 
Teacher Effectiveness in STEM Education 


There is a plethora of research on the belief of Dewey and other constructivists on the impact 
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of teacher effectiveness in students’ learning (Savery, 2006). In addition, there is a wide 
variety of opinions among researchers and educators regarding the effectiveness of teachers 
in instructing STEM education and PBL programs. STEM education is often difficult for 
teachers to implement due to the interdisciplinary nature requiring teachers to combine 
several subjects. The current educational system was designed to teach the core subjects of 
science, technology, engineering, and math in isolation. Teachers were not taught to instruct 
all subjects in conjunction (Lesseig et al., 2017). In addition, Lesseig et al. commented that 
teachers tend to feel overwhelmed and unsure while teaching STEM as often standards being 
taught span grade levels and merge into different subjects that are often not within a teacher’s 
training field. The concerted teaching of all subjects is called integration. According to Wang 
et al. (2011) and many other researchers, integration is most successful when teachers believe 


students are learning effectively and growing in science and mathematics. 


There are many components of teaching that impact teacher effectiveness in teaching STEM 
education. One of the major indicators of teacher effectiveness is how comfortable and 
prepared teachers are in teaching STEM courses (Stohlmann et al., 2012). Gonzalez and 
Kuenzi (2012) reported in a study spanning over 20 years that teachers who earned a college 
degree in mathematics or science had a substantial positive impact on student achievement. 
They stated that in the U.S., most high school teachers do not have a college degree in the 
subject they teach but possess only a degree in secondary education. Most commonly, 
mathematics teachers are the least likely to hold a college degree in mathematics (Gonzalez 


& Kuenzi, 2012). 


Conversely, some researchers disagree and believe that higher teacher content knowledge 
does not directly correlate with greater student achievement and teacher effectiveness. They 
state that teachers who hold a subject-specific degree and demonstrate expertise in that area 
are not necessarily effective teachers. Previous studies cite weak or no correlation linking 
student success and teacher content knowledge (Atkinson, 2012). “It is unrealistic to expect 
teachers to teach or promote engineering when most K-12 teachers do not have a good 
understanding of engineering practices, applications, or careers” (Pinnell et al., 2013, p. 28). 
In addition, most undergraduate teacher education programs do not provide engineering 
principles or practices to prepare teachers for instructing STEM education (Pinnell et al.). 


Currently, there is a lack of effective preservice STEM education for educators which has 
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created a shortage of quality STEM educators at the elementary and middle school level 
(Radloff & Guzey, 2016). 


Collective Teacher Efficacy 


The most important and powerful factor influencing student achievement is the educator’s 
own belief that students can learn and overcome challenges. This phenomenon is called 
teacher collective efficacy, a concept coined by John Hattie in his meta-analysis reviewing 
over 800 studies of factors influencing student achievement (Hattie, 2009). Collective teacher 
efficacy refers to the “collective self-perception that teachers in a given school make an 
educational difference to their students over and above the educational impact of their homes 
and communities” (Tschannen-Moran & Barr, 2004, p. 190). The belief of teachers that 
together they can make a difference outranks all other factors impacting student achievement, 
such as socioeconomic status, home environment, and previous achievement. The influence 
of teacher collective efficacy on student achievement has a reported effect size of 1.57, more 
than double the effect size of other reported impactful factors such as feedback (Hattie, 


2009). 


The importance of teacher collective efficacy and its impact on student achievement is 
pronounced in the STEM arena, as well as the general education setting. In addition, 
individual teachers and their personal beliefs and perceptions that they can grow and impact 
student achievement is crucial to student success. Stohlmann et al. (2012) discussed the 
importance of teachers’ self-efficacy and the belief that they can achieve their goals of 
effectively educating students. A critical component of the development of teacher self- 
efficacy is solid content knowledge and understanding of pedagogy (Er, Artut, & Bal, 2022; 
Kartal & Dilek, 2021; Simsar & Jones, 2021; Stohlmann et al., 2012). Increased student 
motivation, self-esteem, and the development of student self-efficacy is reported when 
teachers feel they themselves are successful as educators. The development of self-efficacy is 
an obstacle among STEM educators as they need more expansive content knowledge and 
advanced pedagogy to provide quality interdisciplinary STEM education (Stohlmann et al., 
2012). The understanding of STEM pedagogy is difficult and can be time consuming, 
particularly for beginning educators (Macalalag et al., 2022; O’Neill et al., 2012). Many 
preservice STEM educators feel ill-prepared to instruct students in the discipline (Radloff & 
Guzey, 2016). 
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Boyd et al. (2009) researched the connection between teacher preparation and student 
achievement and determined, regardless of educational field, the direct link between 
preparation and practice in the first year of teaching is the most successful. Teachers who are 
prepared and determined in their first year in education have the most impact on student 
achievement. In addition, teacher preparation programs focusing on the work of the 
classroom and teachers who actively instruct in the field are impactful for student 


achievement (Boyd et al., 2009). 


There are four sources of creating and shaping collective teacher efficacy beliefs: mastery 
experiences, social persuasion, affective states, and vicarious experiences. The former source, 
mastery experiences is reported to be the most powerful (Bandura, 1977; Bandura, 1986). 
When teachers experience success, such as student achievement attained, they attribute it to 
factors within their control. This increases their collective efficacy which fuels further 
mastery experiences and continues a streak of student achievement (Donohoo, 2009). This 
positive feedback loop creates a snowball effect continuing both increasing the frequency of 


mastery experiences and furthering student achievement. 
Purpose of the Study 


The purpose of this study is to define integrated STEM education within the context of 
middle school and determine the impact STEM programming has on student academic 
achievement compared to their grade level peers participating in a general education setting. 
This study determined the impact of an integrated STEM education program on student 
achievement at the middle school level accounting for other factors affecting student 
performance such as gender, race, socioeconomic status, and attendance rate. The effect of 
integrated STEM programming on student achievement determined the interaction effects of 


demographic factors and student’s attendance rates. 
Research Question 
The research question addressed in this study is to determine the impact of middle school 


integrated STEM programming on student achievement. Currently, little research has been 


conducted on the impact of STEM programming and instruction on student academic 
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performance, particularly achievement. The primary question to be addressed in this research 


is as follows: 


e Does middle school integrated STEM programming positively affect student 


achievement? 


The impact of the following potential moderators on student achievement determined: 


gender, race, socioeconomic status, and attendance rate. 
Objectives of the Study 


The main objectives of this study are as follows: 

1. Define and explain middle school STEM integrated programming. 

2. Determine the interaction effects of factors impacting student achievement of 
middle school students participating in STEM integrated programming. To 
determine the interaction effects of factors impacting student achievement on Ohio 
state assessments in English language arts (ELA), math, and science by use of 
projected proficiency calculations for students participating in integrated STEM 
programming compared to the projected proficiency of general education students. 
The following factors investigated to determine possible interaction effects: 
gender, race, socioeconomic status, and student attendance rate. 

3. Determine the effect of middle school STEM integrated programming on student 
achievement. To test student achievement using Ohio state assessments in English 
language arts (ELA), math, and science for those who participated in middle school 


STEM programming compared to general education students. 
Research Questions 


The research question addressed in this study is to determine the impact of middle school 
integrated STEM programming on student performance and achievement. Currently, little 
research has been conducted on the impact of STEM programming and instruction on student 
academic performance, particularly performance and achievement. The primary research 


question to be addressed in this research is as follows: 
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Does middle school integrated STEM programming positively affect student achievement? 


Additional research questions related to the primary questions include: 


Research Question 1 
Do students participating in an integrated middle school STEM program demonstrate 
differences in academic achievement in ELA, math, and science compared to students 


participating in a traditional general education setting? 


Research Question 2 

Do students participating in the integrated middle school STEM program demonstrate 
differences in achievement due to the interaction effect of gender, race, socioeconomic 
status, or attendance between the STEM program students and general education 


students? 
Methodology 


This quantitative research study may help to determine the impact of integrated STEM 
education on student academic achievement. This was a longitudinal case study analyzing 
student performance on OST over the course of seven years commencing in the 2012-2013 
school year and ending the 2018-2019 school year. First, the definitions of integrated STEM 
education and PBL were described chronicling the PLTW curriculum utilized and other 
STEM-related projects encompassing the middle school STEM program researched in this 
study. Second, student performance was measured using achievement data on OST in math, 
English language arts, and science tests over the course of seven years. Students' OST data 
were analyzed using hierarchical linear modeling (HLM), a type of multilinear modeling used 
to determine the variance among variables in nested data, or data at different hierarchical 
levels. Through the use of HLM, interaction effects of the following demographic factors 
were analyzed to determine their impact on student achievement: student race, socioeconomic 


status, gender, and attendance rate. 
Significance of Study 


The results of this study are highly influential and beneficial to educational administrators 


and policymakers to gain insight into the academic gains of providing STEM programming. 
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This research would provide educational leaders with a clearer view of the impact and 
effectiveness of STEM programming on student achievement. This could assist them with 
making future educational and fiscal decisions regarding the planning, purchasing, and 
implementing of STEM programs, particularly at the middle school level. In addition, this 
research would benefit students as it will help determine the benefits of STEM education 
programs on future student success both on state assessments and on providing mastery 


experiences. 
Role of the Researcher 


The researcher in this study has a solid foundation in math, science, and similar STEM 
subjects and holds a Bachelor of Science degree in Chemistry and a Master of Science degree 
in Inorganic Chemistry. The researcher also earned a Master of Arts degree in Educational 
Administration and Leadership with principal licenses for middle and high school with a 
degree focused on establishing and implementing change. The researcher began her career 
teaching middle school math, science, and STEM courses for three years at the school of 
interest in the study before becoming the kindergarten through 12""-grade STEM instructional 
coach for the school district. The researcher was the district STEM instructional coach for 
several years creating, strengthening, and expanding 21-century initiatives and programs 
from kindergarten to the 12" grade. For the past two years, this researcher has been the 
Supervisor of School Improvement and Curriculum in a medium-sized, urban school district 


in a neighboring county to the school of interest in the study. 
Limitations 


There are a few limitations to this research study. First, the study population is limited to a 
single institution versus data collection from other similar programs in the state. The 
limitation of only including data from one institution prevents generalizability of the 
researcher’s conclusions. Second, student demographic information was self-reported by 
families and parents for several factors such as student race, gender, and socioeconomic 
status. This can limit the validity of the research study. The transition from PARCC to AIR 
assessments for mathematics and English language arts can create reliability issues as two 
different assessments were used to measure student achievement. Other factors seemingly 


unrelated to the STEM program may have effects on student achievement. It is hypothesized 
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that teacher experience and self-efficacy plays an important factor in student learning and 
achievement. The STEM programs studied consisted of very few teachers providing STEM- 
integrated instruction. This makes it difficult to determine the effects of student achievement 
as a function of teacher effectiveness in general or due to the STEM programming and 
curriculum itself. In addition, there were only two integrated STEM teachers for each 
academic year. This can make it difficult to determine if students’ academic achievement was 
related to STEM programming or to the curriculum and instructional materials provided. This 
is connected to the influence of collective teacher efficacy and its impact on student 
achievement. Finally, migration of students in and out of the STEM program may be a threat 
that affects both internal validity and reliability. Fortunately, the migration or number of 
students exiting the STEM Program is limited with very few students exiting the program 


before the end of each academic school year. 
Delimitations 


The delimitations in this research study include the number of participants. The results 
encompassed middle school students attending an urban, traditional school with 
approximately 350 students per grade level and 25 students participating in integrated STEM 


programming within each grade level, respectively. 
Definition of Terms 


Achievement gap- the disparity in academic performance between groups of students, 
particularly in test scores. The achievement gap shows up in grades, standardized-test scores, 
course selection, dropout rates, and college-completion rates, among other success measures 
is most often used to describe the troubling performance gaps between African-American and 
Hispanic students, at the lower end of the performance scale, and their non-Hispanic White 
peers, and the similar academic disparity between students from low-income families and 


those who are more affluent (Ansel, 2011). 
Achievement growth- refers to academic progress made over a period of time measured from 


the beginning to the end of the defined period. The tracking of achievement growth can be 


tracked and determined for individual students, schools, states, or countries, and a wide 
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variety of variables and methodologies may be used to determine whether “growth” is being 


achieved (Achievement Growth, n. d.). 


“All STEM for Some”- A framework of providing STEM education where students in 
elementary, middle, and high school who show the most interest in STEM disciplines are 


fully immersed in STEM education (Atkinson, 2012, Section 1). 


Engineering- is a body of knowledge about the design and creation of products and a process 
for solving problems. Engineering utilizes concepts in science and mathematics and 


technological tools (Honey et al., 2014, p. 14). 


Integrated STEM programming- a meta-discipline encompassing science, technology, 


engineering, and mathematics taught as an integrated whole (Morrison, 2006). 


Mathematics- is the study of patterns and relationships among quantities, numbers, and 


shapes. Mathematics includes theoretical mathematics and applied mathematics (Honey et al., 


p. 14). 


Science- is the study of the natural world, including the laws of nature associated with 
physics, chemistry, and biology, and the treatment or application of facts, principles, 


concepts, or conventions associated with these disciplines (Honey et al., p. 14). 


“Some STEM For All”- A framework for providing STEM education where all students in 
elementary, middle, and high school are introduced to the basic fundamentals of STEM 
education (Atkinson, 2012, Section 1). 


STEM- the study of science, technology, engineering, and mathematics taught as a whole 


with the primary goal of creating critical thinkers for the future workforce (White, 2014). 
Technology- “is the study of the entire system of people and organizations, knowledge, 


process, and devices that go into creating and operating technological artifacts, as well as 


artifacts themselves” (Honey et al., p. 14). 
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Organization of the Study 


This study is organized in the following format: 


Chapter 1 introduces the research problem and provides rationale for the study. The 
introduction, statement of the problem, conceptual framework, basis of STEM 
education, and the purpose of the study are provided. Also included are the 
definitions, assumptions, delimitations, and limitations of the study. 

Chapter 2 contains the review of the literature in reference to the study. This literature 
review provides the definition, history, and an overview of current STEM education. 
Also included are curriculum and standards, an overview of integrated STEM 
education models, impact on student achievement, teacher effectiveness, teacher and 
student self-efficacy, and a review of post-secondary education and career readiness 
in relation to STEM education. 

Chapter 3 describes the research methodology, research model, the subjects, and the 
procedures for the analysis of the data related to student achievement. 

Chapter 4 presents the analysis utilized in the study. These analyses included the use 
of descriptive statistics and hierarchical linear modeling techniques to determine 
differences in achievement in the multilevel data of students, within a classroom, 
within a grade level, within a school. 

Chapter 5 concludes the study with a results’ and discussion section determined from 
the analyses. Recommendations for further research based upon the conclusions are 


also presented. 


Chapter Summary 


This research examined the effect of STEM education and integrated STEM programming on 


student achievement in middle school grades six through eight. The need for this research is 


indicated due to the increase in educational policy and funding to public schools to provide 


STEM programming. There is limited regulation and monitoring of the academic success of 


these programs. The effect of STEM programs on student success both in school and in the 


future workforce is an under-researched area in education. Additionally, there is ambiguity in 


defining integrated STEM education among educators and researchers. The broad definitions 


and effect of STEM programming and PBL on student achievement is a necessary topic of 


research to determine the effectiveness of these increasingly popular and often expensive 
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instructional programs. This quantitative, seven-year longitudinal study determined the 
effectiveness of STEM programming on student achievement, accounting for interaction 
effects of demographic factors, such as student race, gender, socioeconomic status, and 


student attendance. 
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CHAPTER 2: THE LITERATURE REVIEW ON STEM 


EDUCATION 


“It is important that we approach STEM [education] not as a subject, but as a 
mindset.” 


Camsie McAdams (date unknown) 


For the past two decades, the realization of the importance of STEM education in the United 
States is increasingly gaining momentum from kindergarten into high school and beyond 
(Gonzalez & Kuenzi, 2012; Hansen & Gonzalez, 2014; Stieff & Uttal, 2015; White, 2014). 
Dugger (2010) and Hansen and Gonzalez (2014) reported that STEM education brings an 
interdisciplinary approach to traditional schooling, offering students opportunities to view an 
integrated world rather than learning subjects in isolation. The importance of STEM 
education as a method of teaching subjects together to foster innovation was mentioned by 
Barakos et al. (2012) stating, “The quality of modern life depends on innovation and 
development in the STEM disciplines” (p. 1). Researchers and educators have very diverse 
opinions regarding the current success of STEM education today (Gonzalez & Kuenzi, 2012; 
Bybee, 2013). Dugger (2010) stated some view STEM education as an opportunity for 
students to gain important skills necessary to succeed in today’s global economy, while 
others view this method of teaching as interlaced with problems. There are many different 
views regarding best practice methods of providing STEM education and the role of STEM in 


educating students for the future workforce. 


This literature review focuses on the definition of STEM education and provides an overview 
of the varying STEM integration models that are utilized by schools. In addition, methods of 
providing PBL, the history of STEM education, and the different curricula such as PLTW 
will be discussed. Current policy paving STEM education, impact on student achievement, 
curriculum and standards, and the contribution of STEM education to students’ future success 


will be discussed. 
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Defining STEM Education 


The National Science Foundation (NSF) was the first to name STEM education as Science, 
Mathematics, Engineering, and Technology (SMET) with the primary goal of creating critical 
thinkers for the future workforce (White, 2014). According to Sanders (2009), the program 
officer of the NSF commented in the 1990s “SMET” sounded too similar to “smut”, which 
led to the creation of the “STEM” acronym (p. 20). Throughout the past two decades, the 
NSF has funded or assisted with the funding of several projects, such as the Technology for 
All Americans Project (TfAAP) (1994-2005) under the International Technology Education 
Association (ITEA). This created the Standards for Technological Literacy: Content for the 
Study of Technology (STL) and Advancing Excellence in Technological Literacy: Student 
Assessment, Professional Development, and Program Standards (AETL) (Dugger, 2010). 
Both of these projects are influential in the creation of the educational standards and 


foundations existing in STEM education. 


Today the acronym STEM stands for science, technology, engineering, and math with the 


definitions of each subject defined as follows: 


Science- is the study of the natural world, including the laws of nature associated with 
physics, chemistry, and biology and the treatment or application of facts, principles, 


concepts, or conventions associated with these disciplines. 


Technology- is the study of the entire system of people and organizations, knowledge, 
process, and devices that go into creating and operating technological artifacts, as well 


as artifacts themselves. 


Engineering- is a body of knowledge about the design and creation of products and a 
process for solving problems. Engineering utilizes concepts in science and mathematics 


and technological tools. 
Mathematics- is the study of patterns and relationships among quantities, numbers, and 


shapes. Mathematics includes theoretical mathematics and applied mathematics. 


(Honey et al., 2014, p. 14) 
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The individual subjects of science, technology, engineering, and mathematics have learning 
standards by grade level according to the Ohio Department of Education (ODE). For science, 
Ohio’s Revised Science Education Standards and Model Curriculum states, “By the end of 
high school, students should graduate with sufficient proficiency in science to: know, use, 
and interpret scientific explanations of the natural world, generate and evaluate scientific 
evidence and explanations, distinguishing science from pseudoscience, understand the nature 
and development of scientific knowledge, participate productively in scientific practices and 


discourse” (ODE, 2018, p. 3). 


Ohio’s Learning Standards for Technology (ODE, 2017) encompass all technological 
knowledge and skills students in kindergarten through12" grade need to know to succeed in a 
global economy and increasing technological society. The technology standards are separated 
by grade bands: kindergarten through second grade, third through fifth grade, sixth through 
eighth grade, and ninth through twelfth grade, respectively. They are categorized into three 
main disciplines: Information and Communication Technology, Society and Technology, and 


Design and Technology (ODE, 2017). 


Ohio’s Learning Standards for Math (ODE, 2019) describe many areas of expertise that 
educators should strive to accomplish with students. The following are some of the Standards 
for Mathematical Practice: 
make sense of problems and persevere in solving them, reason abstractly and 
quantitatively, construct viable arguments and critique the reasoning of others, model 
with mathematics, use appropriate tools strategically, attend to precision, look for and 
make use of structure, and look for and express regularity in repeated reasoning. (ODE, 


2019, pp. 5-6) 


There are no engineering standards in Ohio for kindergarten through eighth grade, however, 
as part of the Ohio Career Field Initiative, students as young as elementary school can begin 
career planning (ODE, 2019). For students in grades nine through twelve grade there are 
Career Field Technology Content Standards referenced within Engineering and Science 
Technologies and Manufacturing Technologies (ODE, 2019). These engineering standards 
are organized into seven strands: 

- Business Operations/ 21st-century Skills, 


- Electrical/ Electronics, 
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- Computer Integrated Manufacturing, 

- Materials Joining, 

- Pre-Engineering, 

- Design and Development, 

- Precision and Advanced Machining, and 


- Industrial Maintenance and Safety (ODE, 2019) 


There is a vast amount of ambiguity in defining both engineering and technology and in 
determining the appropriate grade-level standards beneficial to student future achievement 


and career success. 
Integrated STEM Education 


The importance of innovation and development of STEM disciplines is associated with a 
higher quality of life, and for this reason, is a recent topic of national conversation (Abdul 
Rahman, Zakaria, & Din, 2021; Barakos et al., 2012; Benek & Akcay, 2019; Chine & 
Larwin, 2022; Hebebci, 2021; Lindsay 2020; Marco-Bujosa, 2021; Preuss et al., 2020; 
Saricam & Yildirim, 2021; Talan, 2021; Williams & Young, 2021)). Stohlmann et al. (2011) 
proclaimed, “Our ever-changing, increasingly global society faces many problems that are 
multidisciplinary, and solving them requires the integration and application of multiple 
science, technology, engineering, and mathematics (STEM) concepts and skills” (p. 32). The 
need for problem-solving and critical thinking skills incorporating the subjects of science, 
technology, engineering, and mathematics is imperative to our nation's economic success and 
has created a collective conversation for the universal definitions describing STEM education 
(Ashford et al., 2016; Asigigan& Samur, 2021; Baharuddin et al., 2021; Batdi, Talan, & 
Semerci, 2019; Bradley 2021; Cetin 2021; Doganca Kucuk et al., 2021; Kim, Park, & Tjoe, 
2021; Minken et al., 2021; Punzalan, 2022; Razi & Zhou, 2022; Seage & Tiiregiin2020; 
Wannapiroon et al., 2021; Yang & Baldwin, 2020). 


The definitions of STEM education for each subject of science, technology, engineering, and 
math clearly demonstrate the ambiguity in what defines STEM discipline. Due to the lack of 
clarity in the definition of STEM education, the term integrated STEM education was 
designed to encompass the discipline as a whole (Giasi, 2018). Honey et al. (2014) defined 


the emergent discipline of STEM integration as interdisciplinary, cross-disciplinary, 
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connected, fused, or transdisciplinary with no definitive boundaries separating each 
discipline. Morrison (2006) referred to integrated STEM education as a “meta-discipline”’ or, 
stated by Kaufman et al. (2003), a “creation of a discipline based on the integration of other 
disciplinary knowledge into a new ‘whole’” (p. 6). According to Stohlmann et al. (2011), 
integrated curriculum, particularly at the middle school level, “provides opportunities for 
more relevant, less fragmented, and more stimulating experiences for learners” (p. 34) further 
stating “integrated mathematics and science has a positive impact on students’ attitudes and 


interest in school, their motivation to learn, and their academic achievement” (p. 34). 


Although there is some ambiguity defining integrated STEM education, there is a clearer 
definition of what it is not, and common misconceptions. According to Morrison (2006) there 
are several false beliefs held by educators and research regarding STEM education methods. 
The first misconception is the belief that technology and engineering are layered into the core 
subject coursework of math and science. Instead of enriching both core subjects, teachers are 
adding and supplementing technology and engineering lessons. The second misconception is 
that technology only refers to adding the use of computers. Morrison (2006) clarified that the 
use of computers does not automatically constitute the use of technology and is not 
necessarily STEM-related. Additionally, technology does not always reference word- 
processing. According to Morrison (2006), another common misconception is that the term 
“hands-on” means active learning is taking place. Hands-on activities do not necessarily 
mean active learning is occurring. Other fallacies pertaining to STEM education listed by 
Morrison (2006) are: the idea that STEM omits laboratory study and the use of the scientific 
method, STEM-educated students must participate in technical fields and do not have the 
knowledge to pursue liberal arts education, math and science education are apart from each 
other, and the common misconception that engineering and technological education is 
difficult and troublesome for students (Morrison, 2006, p. 6).McClure (2017) discussed an 
additional common misconception regarding STEM education: 
The belief that ‘real’ science, technology, engineering, and mathematics learning 
doesn’t occur until children are older, and that exposure to STEM concepts in early 
childhood is only about laying a foundation for the serious STEM learning that takes 
place later. (p. 84) 
The belief that true STEM education does not occur until later years is a hot topic among 
researchers. What is the best time to introduce integrated STEM education models into 


students' formal education? 
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McClure (2017) argued that early STEM programming, particularly from birth to eight years 
old, is just as important as early literacy in the practice of critical thinking, persistence, and 
systematic experimentation. Being naturally born scientists, students are “never too young for 
STEM” (McClure, 2017, p. 84). She argued, “young children can make observations and 
predictions, carry out simple experiments and investigations, collect data, and begin to make 
sense of what they found” (McClure, 2017, p. 84). A two-year research analysis among 
preschoolers determined these young learners can carry scientific practices using the 
scientific method matching that of high schools (McClure, 2017). She further commented that 
early STEM foundations are just as important as early literacy with both of these emerging 
skills predicting future academic achievement (McClure, 2017). Christensen et al. (2015) 
supported the importance of early STEM literacy claiming students are at a critical age in 
their early years for developing attitudes toward STEM education and the exploration of 
future STEM-based careers.On the opposite side of the grade-level spectrum, some 
researchers and experts providing integrated STEM education feel high school is a particular 
impactful time to introduce STEM learning models. Barakos et al. (2012) explained many 
current STEM programs are beginning at the high school level. The North Carolina New 
Schools Project has redesigned over one hundred high schools with the goal of every student 
graduating “ready for college, a career, and life” (Barakos et al., 2012, p. 5). Most of these 
schools have the specific purpose of explicitly teaching high school students using integrated 
STEM instruction, project-based learning, real-life issues, and collaboration. These 
researchers view STEM-focused high schools as the most effective route to generating 
students’ interests in STEM fields and preparing them for STEM-related careers (Barakos et 
al., 2012). 


Recently, inclusive STEM high schools (ISHS) have emerged in states across the country, 
such as California, Massachusetts, Texas, and Ohio. These exclusive STEM-focused 
secondary schools accept students based on interest and not on achievement or aptitude 
(Spillane et al., 2016). Although the practice of choosing students based on STEM interest is 
a powerful method of recruiting invested students, it can further decrease the number of 
females and underrepresented populations. For this reason, many ISHS intentionally recruit a 
larger proportion of minority groups often underrepresented in other STEM-related high 
schools. ISHS are showing promise in “deepening student understanding of STEM, 
bolstering their confidence, and allowing them to see new opportunities for college and 


career” (Spillane et al., p. 59). 
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The argument among researchers is not whether integrated STEM education is effective at 
the elementary, middle, or high school level, but rather which grade level is the introduction 
of STEM practices the most impactful on student achievement and, subsequently, future 
success. Both the elementary and high school years have been shown to be powerful in regard 
to shaping students’ perceptions of their learning and impact on the development of 
integrated STEM education practices. Somewhere between the two may be the “goldilocks” 
zone, the middle school years, when students are beginning to develop attitudes and beliefs 
regarding their abilities in STEM and possibilities of future careers (Christensen et al., 2015). 
Lounsbury (2010, p. 52) recommended integrated STEM curriculum and instruction at the 
middle school level as it “provides opportunities for more relevant, less fragmented, and 
more stimulating experience for learners”, with studies finding the integration of mathematics 
and science having a positive influence on student’s attitudes toward school, their motivation 
to learn, and academic performance. Moreno et al. (2016) remarked, “The middle school 
years are a crucial time for cultivating students' interest in and preparedness for future STEM 
careers” (p. 889). Middle school is a pivotal time for students as their viewpoints on 
education and careers are greatly impacted by their environment and their focus shifts to 


future careers. 


Cohen (2020) stated there are five reasons for introducing integrated STEM education at the 
middle school level. First, student’s academic interest tends to wane in the middle school 
years with many students who enjoy school losing interest in traditional schooling. Integrated 
STEM education revives many students' interest in school subjects. Second, many students 
begin to form career aspirations in the middle school years. The project-based learning 
methods and real-life application involved in integrated STEM education assist students with 
future career exploration. Cohen (2020) stated, “Exposure to STEM careers during this time 
triggers students to seriously consider jobs in engineering, technology, manufacturing, 
biology, etc.” (para. 3). Third, integrated STEM education often facilitates hands-on learning 
which wanes in the middle school years with an increase in long lectures and many subjects 
taught in isolation. Fourth, STEM training teaches problem solving which is particularly 
important in the middle school years as subjects begin to be taught in isolation. Lastly, 
integrated STEM education assists with closing the gender gap by exposing STEM principals 
to both males and females before making definitive decisions regarding future careers 


(Cohen, 2020). 
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Currently, there is no definitive research stating that integrated STEM education is not 
impactful nor important at a certain grade level. However, there are many reasons stating the 
particular importance during the middle school years, otherwise known as the “goldilocks” 
zone. This further supports the exposure of integrated STEM education during middle school 


and a pertinent factor in the proposed research study. 
Project-based Learning (PBL) 


The interdisciplinary and transdisciplinary approach to teaching integrated STEM education 
naturally enforces the use of PBL in the classroom. PBL solidifies the overlap of science, 
technology, engineering, and mathematics (Hansen & Gonzalez, 2014). PBL utilizes many of 
the best practice STEM attributes listed by Morrison (2016), such as creating an active and 
student-centered classroom and serving students with a variety of learning styles. The use of 
project-based instruction allows students to represent, model, and apply their content 
knowledge in interesting and unique ways (Wilhelm, 2014). In addition, Behizadeh (2014) 
stated the increasing importance of PBL to counteract the growing amount of high-stakes 
state assessments and increasing standardization. This allows students to participate in 
authentic instruction and provides an inquiry-based teaching experience for students 
(Behizadeh, 2014). Many researchers agree that PBL learning practices are highly effective 
as a method of teaching STEM education (Behizadeh, 2014; Hansen & Gonzalez, 2014; 
Morrison, 2016). 


Bybee (1997) proposed five aspects of the learning cycle as a roadmap for using PBL as a 
method of instruction of STEM curriculum. Most STEM education curriculums like PLTW 
and EverFi, and PBL programs like the All American Soap use the five aspects of the 
learning cycle developed by Bybee as a foundational learning model derived from 
constructivist theory. The five aspects of the learning cycle theory are: engagement, 
exploration, explanation, elaboration, and evaluation. Engagement involves posing the 
problem to students, ensuring they make connections to previous learning, and pre-assessing 
existing knowledge. During the exploration phase students actively collect data. Explanation 
occurs when students apply their collected data to the research problem, make connections 
with their learning, and bring in new knowledge and vocabulary words. The fourth phase, 
elaboration, involves students presenting and actively applying their new learning to the 


research problem. This occurs in conjunction as the teachers present new challenges which 
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require critical thinking to solve. Lastly, the evaluation phase assesses students’ learning, not 
necessarily in a pencil-paper test, but through presentation of the research problem and 
solution and final outcomes of the project (Bybee, 1997). Bybee’s (1997) proposed five-step 


PBL model is used today in many STEM activities and purchased curriculums. 
History of STEM Education 


There are many historical events that led to the development of STEM education. The Morrill 
Act of 1862, which granted land to universities for the development of agricultural training, 
eventually spawned the development of the first engineering-based training programs. For 
example, Ohio State University was initially called the Ohio Agricultural and Mechanical 
College (White, 2014). According to White (2014), land grant establishments began to 
develop and the concept of engineering based training formed into career and technology 


training and the creation of vocational high schools. 


The other two historical events discussed by Gonzalez and Kuenzi (2012) and Hansen and 
Gonzalez (2014) that facilitated the development of STEM education were World War II and 
the Soviet Union’s launch of Sputnik 1. World War II brought increased research into 
advancing technologies with academic researchers, scientists, and the military working in 
collaboration to aid in the war effort and consequently propelling STEM education (Gonzalez 
& Kuenzi, 2012). The Soviet Union’s launching of Sputnik 1 in 1957 began the Space Race. 
The United States and the Soviet Union were suddenly in competition to win the lead in the 
advancement of worldwide technology, STEM education, and, ultimately, global control 
(Gonzalez & Kuenzi, 2012; White, 2014). Gonzalez and Kuenzi (2014) reported that this 
scared the United States by motivating them to push for technological advancement and the 
improvement of STEM education, with Congress passing the “Space Act” in 1958 forming 
NASA and the federal government’s response with the formation of NDEA. The mission of 
NASA was to “expand and improve” the presence of the United States in space and 
maximize the use of science and engineering to complete that endeavor (White, 2014, p. 3). 


All of these events were pivotal in the creation and sustainment of STEM education. 


In 1983, A Nation at Risk was published as an educational reform policy reviving the STEM 
movement in public education (Mahoney, 2009; Mahoney, 2010). Mahoney (2010) reported 


A Nation at Risk was extremely influential in the development of national standards produced 
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by the National Council of Teachers of Mathematics (NCTM), the National Research Council 
(NRC), the International Technology Education Association (ITEA) and the American 
Association for the Advancement of Science (AAAS). These interest groups were pivotal in 
the advancement of science, technology, engineering, and math both as individual subjects 


and combined as STEM education (Mahoney, 2010). 


In addition, the No Child Left Behind (NCLB) revived legislation of the Elementary and 
Secondary Education Act (ESEA) and reinstated Title 1 including disadvantaged students to 
the provisions. NCLB supported educational reform through standards-based education 
following a path of high standards and goals measured through state assessed basic skills. 
The caveat to receive federal funding under NCLB and, subsequently, Title 1 is that all 
schools must administer state assessments to students in certain subjects and specific grade 


levels (U.S. Department of Education, 2004). 


In 2015 the U.S. passed the Every Student Succeeds Act (ESSA) taking away some control 
from the federal government in public education evident in NCLB. Under this renewed 
federal law, more power and decision-making latitude were given to local school districts and 
states, including the use of funding, particularly under Title IV (Achieve, 2017; ESSA, 2017). 
The passage of ESSA has given local school districts the abilities and funding to provide 


formal STEM programming, digital literacy curriculum, and the purchasing of technology. 


In 2019, two bills were passed through Congress further enhancing public education’s ability 
to provide STEM programming to meet the needs of all students, including underrepresented 
populations. The STEM Research and Education Effectiveness and Transparency Act was 
passed in January 2019 allowing for broadening participation in National Science Foundation 
(NSF) research and educational programs including STEM education (STEM Research and 
Education Effectiveness and Transparency Act, 2019). The second bill, the 2/st Century 
STEM for Girls and Underrepresented Minorities Act, passed in May 2019, provides grants 
to local educational agencies to empower girls and other underrepresented minorities to 
participate in STEM programs and pursue STEM field careers (21st Century STEM for Girls 
and Underrepresented Minorities Act of 2019, 2019). All of these events and policies have 
shaped and modeled STEM education into what it is today. 
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STEM Education in Ohio, Pennsylvania, and West Virginia 


Many economic and geological factors shaped the state of STEM education in the area 
known as the Appalachia Partnership Initiative (API) area. The API area consists of 27 
counties within Ohio, Pennsylvania, and West Virginia known for producing the majority of 
natural gas in the nation and the primary location of the Marcellus and Utica Shales. 
Ironically, this area known as the “rust belt” saw significant economic decline and 
deindustrialization beginning in the 1970s due to the decline in steel production (Crandall, 
1993). However, with the resurgence of natural gas through hydraulic fracking in the API 
regions the “rust belt” is in an economic upswing. Due to the vast amount of fossil fuels and 
STEM-related labor and manufacturing jobs in the API region, there is a tremendous need for 
STEM education and workers. In 2017, the Rand Corporation completed a report on 
employment in energy and advanced manufacturing—related industries and on various 


indicators of effective STEM education (Gonzale et al., 2017). 


The majority of natural gas production occurs within two geographical regions: the Marcellus 
Shale that includes Ohio, Pennsylvania, West Virginia, New York, Maryland, and the Utica 
Shale, with the majority of production in Ohio, Pennsylvania, and West Virginia, respectively 
(Gonzalez et al., 2016). This has created a significant rise in the number of middle-skilled 
workers needed to fill positions in the energy industry (Gonzalez et al.) with the nation 
unable to fill many of the current STEM-related positions (Atkinson, 2012; Gonzalez & 
Kuenzi, 2012; Gonzalez et al., 2017). 


The Appalachia Partnership Initiative (API), launched in 2014 by the Social Investment 
Team of the Chevron North America Appalachian/Michigan Business Unit, was created with 
the primary and long-term goal of preparing and training local workers for careers in the 
energy and advanced manufacturing regions within the Marcellus and Utica Shale areas 
(Gonzalez et al., 2016; Gonzalez et al., 2017). The API is devoted to investing $20 million 
into STEM education from kindergarten to 12'" grade dispersed among 27 counties in Ohio, 
Pennsylvania, and West Virginia (Gonzalez et al., 2016). According to Gonzalez et al. 
(2016), they are also committed to developing STEM workforce development programs 
along with other stakeholders, such as public schools, training institutions, industry, and 


nonprofit entities. This is in order to increase preparedness, hands-on training, and skills 
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necessary to meet the demands of jobs in the energy and advanced manufacturing fields 


(Gonzalez et al., 2016). 


The API has collected a considerable amount of data related to STEM education, workforce 
development, wages, and employment trends all gleaned internally or from outside sources, 
such as the U.S. Census Bureau, National Assessment of Educational Progress (NAEP), the 
U.S. Department of Education’s National Center for Educational Statistics Integrated 
Postsecondary Education Data System (IPEDS), and the Department of Education for each 
state (Gonzalez et al., 2016; Gonzalez et al., 2017). There were four key findings reported by 
the API that are important given the scope and research of this literature review. First, 
although many counties within the API region are decreasing in population, the overall 
number of people in the 27 county region is unchanged due to significant growth in a number 
of counties. Second, workers in STEM fields received the highest pay in the regions. 
However, the average pay for STEM workers in the region was below the national average 
for STEM workers with the same education and experience. Third, the API states of Ohio, 
Pennsylvania, and West Virginia, saw an increase in graduation rates matching the national 
trends. West Virginia, however, was behind the national averages in student math and science 
achievement. Fourth, the Rand Corporation determined that institutions of higher education 
were educating students in majors able to fill the needs of local employers in the STEM 


workforce (Gonzales et al. 2016; Gonzalez, et al., 2017). 


According to Gonzalez et al. (2017), the main conclusion from the API research was that the 
region is producing and sustaining local STEM workers higher than the national average. 
However, they did find the number of workers able to work as determined by age is 
decreasing. They also found Ohio and Pennsylvania graduate more high school students who 
enter STEM fields in their careers or college majors than the national average, with West 
Virginia lagging behind. The API determined Ohio, Pennsylvania, and West Virginia are 
ahead of the nation in preparing students to enter the workforce. They are also leading the 
nation in the percentage of STEM-related associate’s degrees and certificate programs. In 
addition, they found the number of workers of working age is decreasing with no evidence 
that there is enough supply to meet the upcoming and current demand of jobs in the energy 


and advanced manufacturing fields (Gonzalez et al., 2016; Gonzalez et al., 2017). 
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Funding for STEM Education 


Along with the API, there are several funding sources for a number of recent STEM 
education initiatives. Gonzalez’s and Kuenzi’s (2012), Specialist in Science and Technology 
Policy and Specialist in Education Policy, respectively, reported four inventories in STEM 
education mandated by Congress with “three of the four agencies making up four-fifths of 
federal funding in STEM education. These three agencies are: the NSF, the Department of 
Education (ED), and the Department of Health and Human Services (HHS)” (p. 4). A third of 
all federal funding reserved for STEM education is appropriated to the NSF with the majority 
of the funding spent on student degree attainment, research, and the development of STEM 
careers (Gonzalez & Kuenzi, 2012). The National Science Foundation’s Innovative 
Technology Experiences for Students and Teachers (ITEST) program has continued to 
support kindergarten through 12"*grade students and teachers in providing STEM-based 
activities with the goal of creating STEM career pathways (Connors-Kellgren et al., 2016). 


The previously discussed renewed ESSA Act in 2015 gave more power, funding, and 
decision-making abilities to local school districts and states, particularly under Title IV 
(Achieve, 2017; ESSA, 2017). Title IV provides funding for 21*-century initiatives, Student 
Success and Academic Enrichment (SSAE) Grants, and after school enrichment programs. 
According to ESSA, beginning in 2017, $1.6 billion were given to districts under the SSAE 
Grants with the intention of improving student academic achievement. Using similar funding 
determinations as Title I, districts can use the support to improve the use of technology, 
including STEM education and digital literacy to increase student achievement (ESSA, 
2017). Under ESSA, there are several activities and programs supported by Title IV with the 
intention of providing students with a well-rounded education. ESSA supports the expansion 
of high-quality STEM courses and increases access to STEM students in at-risk and 
underserved groups. ESSA also ensures that students get support in nonprofit STEM 
competitions and programs allowing for hands-on opportunities for students. The act supports 
the integration of other academic subjects, such as social studies, as well as the arts into 
STEM programs. ESSA seeks to support the integration of informal, integration programs 


and afterschool STEM programs also assisting with environmental education (ESSA, 2017). 


The increased funding for 21‘-century programming, including STEM education, provided 


by the passage of ESSA has created a demand for high-quality STEM curriculum and 
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programming. The passage of ESSA has sparked national interest in the STEM disciplines. 
This has created the realization of the importance of development and innovation of STEM 
education, making this a critical component of recent educational reform. In addition, the 
support from ESSA for the integration of content subjects, such as science and mathematics 
into both formal and informal STEM instruction has further promoted the push for integrated 


STEM education (Achieve, 2017). 
Post-secondary Education and Career Readiness 


Gonzalez and Kuenzi (2012) reported more than half of federal funding reserved for STEM 
education is appropriated for undergraduate, graduate, and career-based training education 
and research. A considerable amount of this funding is used for remediation programs and to 
increase retention rates in STEM fields and career-based programs (Gonzalez & Kuenzi, 
2012). The primary motivation of remedial programs is to fill the demand for jobs in the 


STEM fields, such as mathematicians, scientists, and engineers. 


The majority of students in STEM programs at the post-secondary and college level continue 
to perform at or below basic proficiency levels (Stieff & Uttal, 2015). According to Gonzalez 
et al. (2017) and Rand Corporation research, higher education institutions are doing an 
effective job at preparing college students for the local STEM workforce. However, Han and 
Buchmann (2016) reported half of all college students majoring in a STEM field leave the 
field before graduation and pursue a different degree field. The lack of perseverance and 
academic underperformance of STEM students at the college level has created a need for 
research regarding the academic performance and achievement of students in middle and high 


school. 

Implications for Educational Leaders 

There are many implications for educational leaders regarding their role in the development 
and implementation of STEM programming. The role of educational leaders has become 


increasingly significant due to the recent political power shifting to local school districts and 


increased funding, such as those mandated by ESSA (Achieve, 2015; ESSA, 2015). 
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Research on the challenges of STEM education reform at the kindergarten through 12""grade 
level most often focus on the important role of teachers and students (National Research 
Council, 2011). The beliefs and attitudes of teachers and students are found to be most 
influential (McGinnis et al., 2002). Many studies regarding the impact of collective teacher 
efficacy and student self-efficacy refer to the strong influence of teacher and student beliefs 


as an indicator of student achievement and success (Donohoo, 2007; Hattie, 2009). 


Although teacher and student attitudes and beliefs play an important role in shaping all 
general education reform, Bybee (2013) discussed the major challenges specific to STEM 
education. He stated these are related to the misunderstandings surrounding the definition of 
STEM education and the qualifications of STEM literacy. The themes related to reform 
according to Bybee (2013) include: focusing on global challenges with citizen understanding, 
changing societal views of environmental issues, identifying 21*-century skills in the 
workforce, and challenges facing national security. School leaders play an important and 
foundational role in ensuring the implementation and sustainability of STEM reform (Bybee, 


2013; Waight et al., 2018). 


Expanding on the prior research of Bybee (2013), Waight et al. (2018) identified five main 
components regarding the role of educational leaders tackling STEM reform. First, leaders 
must engage in active student recruitment for STEM programs. Second, educational leaders 
must facilitate school and community-based activities. Next, leaders must make students 
aware and prepare them to contribute to the local economy. Additionally, they must solve 
administrative and logistical issues. Lastly, educational leaders must provide effective 
professional development for teachers. All of these themes of an effective educational leader 


are imperative to the success of STEM reform and program development (Waight et al.). 


In 2010, President Obama’s Council of Advisors on Science and Technology (PCAST) 
prepared a report with five recommendations for educational leaders and stakeholders. These 
recommendations are necessary to improve STEM education with the main goal of preparing 
and inspiring future students into joining STEM fields (Kennedy & Odell, 2014). There are 
five recommendations by PCAST and they are described by Kennedy and Odell (2014). The 
first recommendation is to improve federal coordination and leadership on STEM education. 
This is an important implication for educational leaders previously discussed by Waight et al. 


(2018). The second recommendation is for leaders to support the state-led movement to 
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create a common benchmark for student learning in STEM. This is related to the creation of 
national standards and curriculum that is needed in STEM education. Third, is to foster, 
recruit, and reward teachers for inspiring and preparing students to pursue STEM careers and 
higher education. The fourth recommendation for leaders is to create STEM-related 
experiences to increase students’ interest. Fifth, policymakers and stakeholders need to 
support states and local school districts in creating STEM education reform (PCAST, 2010). 
According to Kennedy and Odell (2014), the purpose of these recommendations for 
educational leaders and stakeholders is to ensure achievement in three goals of K-12 STEM 
education; to expand the STEM workforce, to increase technological and STEM literacy, and 


to improve advanced training and career preparedness (p. 249). 
Impact on Student Achievement 


The impact of STEM education policies and initiatives on student achievement report varying 
degrees of success (Dugger, 2010; Gonzalez & Kuenzi, 2012; Snyder, 2018; White, 2014). 
Gonzalez and Kuenzi (2012) attested there is no single statistic that can fully quantify or 
encompass the condition of STEM education in the nation and for a variety of reasons the 
question “what is the condition of STEM education?” may be unanswerable” (p. 9). Many 
researchers and experts note that it is difficult to determine the extent STEM education has 
impacted students. Although, from a broad perspective, STEM education has maintained or 
improved its impact over the past 40 years. Gonzalez and Kuenzi (2012) commented that it is 


difficult to measure the success of the United States educational system due to its complexity. 


In 2019, the most comprehensive report of the current state of student achievement in relation 
to STEM education was published by the National Science Board, titled Science and 
Engineering Indicators 2020. This federally funded government report found the United 
States ranks in the middle among 19 advanced nations in student performance in the STEM 
fields of science, technology, engineering, and mathematics. STEM students in the U.S. are 
underperforming compared to students in other developed countries such as Singapore, 


Taiwan, and South Korea (National Science Board, 2019). 


Nationally, U.S. students’ achievement in mathematics has increased in the last 30 years with 
the largest growth occurring in the first two decades (National Science Board, 2019). Figure 1 


illustrates students’ academic achievement in mathematics measured by performance on the 
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National Assessment of Educational Progress (NAEP), the largest nationally recognized 
assessment used for representative sampling to determine performance of all students 
nationally. Average mathematics scores of 8'"-grade students have grown since 1990 with 
improvement slowing in the past 10 years with an average mathematics score of 281 in 2007 


and 283 in 2017, respectively (National Science Board, 2019). 


Yea! 


Figure 1. Average NAEP Mathematics Scores of Students in Grade 8: 1990-2017, 2019 
Note. The scale for NAEP mathematics assessment scores is 0-500 for grade 8 


Student mathematics achievement as evidenced by NAEP scores vary greatly according to 
student demographics. Figure 2 displays the important findings related to gender, student 
race, parent educational level, and socioeconomic status (SES). Male students outperform 
females with an average score, in 2017, of 283 for males versus 282 points for females. 
Although this appears as a small difference, it is statistically significant given such a large 
sample size with greater than 11,000 students per year (National Science Board, 2018). 
Student race shows large disparities in mathematics achievement with Asian or Pacific 
Islander students earning 310 points and White students achieving 293 points. Hispanic 
populations averaged 269 points. Lastly, Black or African Americans earned only 260 points 
on average, some 50 points behind the achievement in the Asian and Pacific Islander 


population (National Science Board, 2019). 
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Figure 2. Average Scores of Students in Grade 8 on the NAEP Mathematics Assessment, by 
Student Characteristics: 2017, 2019 


Note. The scale for NAEP mathematics assessment scores is 0-500 for grade 8. Hispanic may 


be any race; race categories exclude Hispanic origin 


According to the National Science Board (2019), students whose parents graduated from 
college showed statistically higher achievement than students whose parents only graduated 
from high school. This large difference in math performance was also grossly evident 
according to SES among students. SES was determined by the eligibility of students in the 
National School Lunch Program (NSLP). The achievement gap between students in the low- 
SES category has been consistently 26-30 points below students in the high-SES category for 
the entire duration of the 21-year study. 


Figure 3 illustrates the achievement gap between students eligible for free or reduced-price 


lunch (low-SES) and not eligible for free or reduced-price lunch (high-SES). 
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Year 


~o Eligible for free or reduced-price lunch ~- Not eligible for free or reduced-price lunch 


Figure 3. Average NAEP Mathematics Scores of Students in Grade 8, by Eligibility for 
National School Lunch Program: 1996-2017, 2019 


Note. NAEP uses eligibility for the federal National School Lunch Program (NSLP) as a 


measure of socioeconomic status 


NSLP is a federally assisted meal program that provides low-cost or free lunches to eligible 
students. It is sometimes referred to as the free or reduced-lunch program. Information on 


students’ eligibility for free or reduced-price lunch was first collected in 1996. 


There is a significant achievement gap in NAEP mathematics scores among various 
underrepresented populations as evidenced by the National Science Board (2019). To 
determine technological and engineering literacy, the National Science Board administered 
the NAEP Technology and Engineering Literacy Assessment with students’ average 
performance increasing every year since 2014. However, little subgroup data on 
underrepresented populations have been conducted on technology and engineering 
achievement (National Science Board, 2019). In order to gain a better perspective into the 
state of STEM education, this study will analyze performance and academic achievement for 
subgroups of students such as women, minorities, and other underrepresented populations 


compared to the general education population. 
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STEM Careers and Workforce Development 


The National Science Board (2019) concluded students taking high school STEM courses or 
who had exposure to STEM-related programs were more likely to get a STEM-related job in 
the workforce directly after high school or take technical education courses to gain the skills 
to get a technical job. However, similar to academic achievement, there are large disparities 
among underrepresented groups participating in STEM careers and in the workforce 
(Gonzalez & Kuenzi, 2012; Hebebci, 2019; Kennedy & Odell, 2014; National Science Board, 
2019; Funk & Parker, (2018a). 


Underrepresented Groups 


An ongoing trend reported by many researchers is the increasing achievement gaps among 
underrepresented groups such as minorities, women, and economically disadvantaged 
students (Beede et al., 2011; Gonzalez & Kuenzi, 2012; Kennedy & Odell, 2014; Sanders, 
2009; National Science Board, 2019). Gonzalez and Kuenzi (2012) stated that these gaps in 
STEM achievement are evidenced by several data sources, such as test scores, degree 
acquisition, and employment in STEM-related careers. Gonzalez and Kuenzi (2012) argued 
underrepresented groups are an untapped resource for potentially filling many STEM-related 
positions currently needed in the workforce. They explained that the exposure to STEM and 
increase in participation in STEM fields among women and racial minorities may pose a 
solution to the labor shortage issue described by Kennedy and Odell (2014) and Gonzalez et 
al. (2017) in their review under the API. 


Contrarywise, Gonzalez and Kuenzi (2012) and Kennedy and Odell (2014) cited that science 
and engineering enrollments have grown by 35% in the past decade with Hispanic, American 
Indian, and African American students increasing by 65%, 55%, and 50%, respectively. In 
2021, the largest increase in U.S. student enrollment is expected to occur among bilingual 
Hispanics, necessitating a future demand for bilingual STEM education (Kennedy & Odell, 
2014). The study of underrepresented populations within STEM fields is a popular area of 
research. Banning and Folkestad (2012) concluded in their meta-study reviewing 101 
dissertations that recruitment and retention of underrepresented students in STEM-related 


fields was a major topic among researchers over the span of their 20-year study. 
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In more recent years, inclusive ISHS in Texas, North Carolina, Tennessee, and Ohio have 
developed with the main focus of recruiting and retaining higher ratios of underrepresented 
students, particularly African-Americans, economically disadvantaged students, women, 
Hispanics, and first-generation college students (Spillane et al., 2016). The mission of ISHS 
is to prepare students in underrepresented populations, focusing on minorities, for success in 
STEM-related fields and college majors (Lynch et al., 2018; Spillane et al.). Ultimately, ISHS 
assist with lowering the achievement gap among minorities and other underrepresented 


populations (Lynch et al.; Spillane et al.). 


Despite the increased demand to fill jobs in the STEM fields, such as mathematicians, 
scientists, and engineers, the majority of students in STEM programs at the post-secondary 
and college level continue to perform at or below basic proficiency levels (Stieff & Uttal, 
2015). Students in middle school, nationally, are underperforming as evidenced by the 
majority of eighth grade students scoring below basic levels on the NAEP science test. 
Researchers conclude that U.S. students are receiving inadequate academic instruction in 
STEM education (Etim, Etim, & Blizard, 2020; Stohlmann et al., 2011; National Science 
Board, 2019). Although the demand for STEM professions is at an all-time high, the number 
of STEM college graduates is low and not meeting the national demand with not enough 
students choosing to major in a STEM field and even disproportionately less minorities and 
women are choosing the career field (Doerschuk et al., 2016; Gonzalez & Kuenzi, 2012; 
Funk & Parker, (2018a). The underperformance of STEM students beginning in kindergarten 
through high school and beyond has created a need for further research regarding academic 


performance and achievement. 
Gender 


A decade ago, it was reported that women earned more college degrees than men, however, at 
the time they also held less than one quarter of the STEM-related jobs in the nation (Beede et 
al., 2011; Gonzalez & Kuenzi, 2012). More recently, Funk and Parker (2018a) announced the 
share of women working in STEM occupations had increased and is about half, matching 
men in the number of STEM workers. Figure 4 displays the representation of women in 
STEM jobs illustrating the gender differences in STEM careers varies greatly depending on 
the area of STEM career. 
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Figure 4. Representation of Women in STEM Jobs Varies Widely 


Note. Based on employed adults’ ages 25 and older. Each circle represents a single 


occupation (e.g., mechanical engineer, registered nurse). Engineering includes architects. 


Women make up 96% of speech language pathologists and 95% of dental hygienists, 
respectively. Conversely, women encompass less than 10% of sales engineers and only 8% of 
mechanical engineers (see Figure 4). For computer occupations, which has experienced the 
greatest growth in available jobs in recent years, women’s representation has decreased from 
32% in 1990 to 25% in recent years (Funk & Parker, 2018a). The representation of women in 
STEM careers appears highly dependent upon the STEM job cluster. The causes for the 
underrepresentation of women among particular STEM job clusters is debated among 
researchers (Beede et al., 2011; Gonzalez & Kuenzi, 2012; Shapiro et al., 2015). Some state 
the reasons are due to self-efficacy, discrimination, school culture, and the bias of girls 
participating in science (Gonzalez & Kuenzi, 2012). Others attribute this phenomenon to the 
influence of family and upbringing, with women continuing their roles as primary caregivers, 
maintaining society's expectations influencing career choices (Gonzalez & Kuenzi, 2012), a 
lack of women role models, gender stereotyping, and a more difficult work/life balance in 
STEM career fields (Beede et al.). Shapiro et al. were surprised by these findings because 


girls currently perform better academically than boys in high school with average GPAs of 
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3.42 and 3.28, respectively. Shapiro et al. called the decreasing number of girls, as they 
advance through school participating in leadership positions and STEM fields, the “leaky 
pipeline” (Shapiro et al., p. 3). The leaky pipeline of women into leadership roles and STEM- 
related careers is evident with women taking on more traditional female-dominated, or “pink 
collar” jobs, such as nursing, education, or retail sales (Maltese & Tai, 2011, p. 877; Shapiro 
et al., p. 3). Although described in different ways, this phenomenon is evidenced among 
many researchers (Beede et al.; Gonzalez & Kuenzi, 2012; Maltese & Tai, 2011; Funk & 
Parker, 2018a; Shapiro et al.; Spillane et al., 2016). 


Minority Students 


Black and Hispanic students are greatly underrepresented in STEM careers and jobs in the 
workforce in relation to all U.S. occupations. In 1990, Blacks made up 7% of STEM workers 
and increased to a mere 9% by 2018. Hispanics filled just 4% to 7% in the same time period. 
In contrast, Asians are overrepresented in relation to the number of total workers with over 
17% being both college-educated and STEM workers. Figure 5 illustrates the 


underrepresentation of Blacks and Hispanics across many clusters of STEM careers. 


% of employed in each occupational group who are ... 
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All employed 
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Figure 5. Blacks and Hispanics Underrepresented across Most STEM Job Clusters 
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Note. Based on employed adults’ ages 25 and older. Whites, Blacks, Asians include only non- 
Hispanics. Hispanics are of any race. Other and mixed-race non-Hispanics are not shown. 


Engineering includes architects. 


According to the Funk and Parker (2018b), health technicians and nursing positions have the 
largest proportion of Black and Hispanic workers. Thirty-seven percent of medical 
technicians and vocational nurses are either Black or Hispanic, while only 17% of registered 
nurses are Black. The most recent data state the percent of STEM workers is as follows (See 
Figure 5): White (69%), Asian (13%), Black (9%), and Hispanic (7%). There is a large 
disparity between different STEM job clusters among minorities (Funk & Parker, 201 8b; 
Gonzalez & Kuenzi, 2012). 


The underrepresentation of minorities in the STEM workforce is a direct byproduct of the 
achievement gaps evident among minorities in kindergarten through high school graduation 
(Gonzalez & Kuenzi, 2012). Gonzalez and Kuenzi (2012) commented researchers have 
identified dozens of variables responsible for the achievement gap in STEM-related fields 
among minority populations, such as lack of parental involvement, lower socioeconomic 
status (SES), “a lack of resources (underfunding) and less qualified teachers at schools that 
serve minority students, teachers’ low expectations, stereotype threat, and racial oppression” 
(p. 24). They reported at least a 20-point decrease in African American and Hispanic students 
compared to White students on the 4'" and 8'"grade NAEP in 2011 (Gonzalez & Kuenzi, 
2012) and in 2017 (National Science Board, 2019). The 2017 Review, produced by Gonzalez 
and Kuenzi (2012), lists many factors that positively and negatively influence minority 
students’ preparedness in STEM education. The positive factors are: parental support and 
involvement, early age introduction to STEM, self-efficacy in STEM subjects, exposure to 
STEM-related activities, and opportunities for bilingual STEM education. The negative 
factors include: lack of resources due to underfunding, teachers in schools with a large 
proportion of minority students under-qualified in STEM fields, limited advanced placement 
classes, low expectations from teachers, bias and stereotypes among minorities, and a higher 
dropout rates (Gonzalez & Kuenzi, 2012). The many factors responsible for the achievement 
gap in STEM education among minorities is a complex, multi-faceted issue fueling much of 


the research within this study. 
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Curriculum and Standards 


Currently, in the U.S., there are no nationally-recognized standards for STEM education, nor 
are there any for the state of Ohio. Han and Buchmann (2016) stated the lack of curriculum 
standardization in the U.S. has contributed to lower science achievement and most likely 
other disciplines in STEM, as well. The lack of continuity and standards in STEM education 
has stimulated the National Research Council (NRC) and the National Academies of 
Engineering to highlight the need for intentional and explicit instruction of STEM curriculum 


and standards (Radloff & Guzey, 2016). 


Despite the lack of current STEM standards, there are Standards for Technological Literacy 
(STL) developed by the ITEA, 2000/2002/2007). The most recent STL version was released 
in 2007. The ODE released the 2017 Ohio Learning Standards in Technology, updating the 
2003 learning standards. This divided technological education into three main disciplines: 
Information & Communication Technology, Society & Technology, and Design & 


Technology (ODE, 2019). 


Many non-profit providers have developed STEM curriculum intended to be used in public 
education. Two of these are reviewed in literature: PLTW and The Infinity Project 
(Stohlmann et al., 2012). PLTW, the nation’s largest non-profit STEM curriculum for middle 
and high schools is increasingly used today in classrooms but little research has been 
conducted on its impact on student achievement and future career choices (Berland, 2013). 
Lawanto et al. (2012) researched the relationship between interest and success expectancy in 
STEM careers for students taking a PLTW Engineering Design course. Lawanto et al. 
determined there was a significant relationship between student self-interest in engineering 
design and their expectancy for success with intrinsic motivation being a predictor of future 


success and choice of STEM career. 

Project Lead the Way 

Due to the increase in national funding allocated to public schools, particularly under ESSA 
(2015), there has been an influx of engineering and STEM curriculums designed for middle 


school. Currently, the nation’s leading STEM curriculum for middle and high school is 


Project Lead The Way (PLTW, 2020). PLTW is a nonprofit organization with the goal of 
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providing an impactful interdisciplinary STEM curriculum for students in grades 
kindergarten through 12" grade (Stohlmann et al., 2011). PLTW focuses on STEM 
curriculum comprising areas of computer science, engineering, and biomedical science with 
the intention of improving problem solving, critical thinking, communication, perseverance, 
and creativity. PLTW is aligned to the Next Generation Science Standards and the Common 


Core State Standards for ELA and Math (PLTW, 2020). 


The middle school program, PLTW Gateway, consists of 10 units designed to be taught daily 
for nine weeks (PLTW, 2020). Below are the 10 PLTW Gateway units with course 


descriptions: 


Design & Modeling- Students discover the design process and develop an 
understanding of the influence of creativity and innovation in their lives. They are then 
challenged and empowered to use and apply what they have learned throughout the unit 


to design a therapeutic toy for a child who has cerebral palsy. 


Automation & Robotics- Students learn about the history and impact of automation 
and robotics as they explore mechanical systems, energy transfer, machine automation, 
and computer control systems. Using the VEX Robotics platform, students apply what 


they know to design and program traffic lights, robotics arms, and more. 


App Creators- This unit exposes students to computer science as a means of 
computationally analyzing and developing solutions to authentic problems through 
mobile app development and will convey the positive impact of the application of 


computer science to other disciplines and to society. 


Computer Science for Innovators and Makers- Throughout the unit, students learn 
about programming for the physical world by blending hardware design and software 
development, allowing students to discover computer science concepts and skills by 


creating personally relevant, tangible, and shareable projects. 


Energy and the Environment- Students are challenged to think big and toward the 
future as they explore sustainable solutions to our energy needs and investigate the 


impact of energy on our lives and the world. They use what they have learned to design 
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and model alternative energy sources, as well as evaluate options for reducing energy 


consumption. 


Flight and Space- The exciting world of aerospace comes alive through Flight and 
Space Students explore the science behind aeronautics and use their knowledge to 


design, build, and test an airfoil. 


Science of Technology- Science impacts the technology of yesterday, today, and the 
future. In this unit, students apply the concepts of physics, chemistry, and 
nanotechnology to activities and projects, including making ice cream, cleaning up an 


oil spill, and discovering the properties of nano-materials. 


Magic of Electrons- In this unit, students examine the behavior and parts of atoms as 
well as the impact of electricity on the world around them. They learn skills in basic 
circuitry design and use what they know to propose designs such as burglar alarms for 


an art museum. 


Green Architecture- In this unit, students learn how to apply green concepts to the 
fields of architectural sustainability and apply what they have learned to design 


affordable housing using Autodesk’s 3D architectural design software. 


Medical Detectives- Students play the role of real-life medical detectives as they 
collect and analyze medical data to diagnose disease. They solve medical mysteries 
through hands-on projects and labs, measure and interpret vital signs, examine nervous 
system structure and function, investigate disease outbreaks, and explore how a 
breakdown within the human body can lead to dysfunction. 


(PLTW, 2020) 


All 10 PLTW Gateway units were taught as nine-week units in this study except for App 
Creators and Science of Technology. PLTW Gateway curriculum is intended to be taught 
along with the many components discussed earlier by Honey et al. (2014) as interdisciplinary, 
cross-disciplinary, and connected with no thorough boundaries separating each discipline. 
Morrison (2016) discussed several suggested attributes of an effective STEM classroom that 


follow the principles of PLTW for grades six through 12: active and student-centered, 
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supportive of various modalities of learning, accommodating of many different learning 
styles and able to serve students with disabilities. Many of these attributes are also discussed 


by Bybee (1997) in his five steps of the learning cycle of PBL. 


In addition, there are several technical and equipment-related attributes necessary for a 
successful STEM classroom (Morrison, 2016). These attributes are needed to ensure success 
in facilitating many of the PLTW Gateway modules discussed: classroom and laboratory are 
physically one, classroom furniture is easily moved and reconfigured, small tools and 
manipulatives are available, specific equipment readily available to support inquiry-based 
learning, and computers with the necessary software are readily available for students 


(Morrison, 2016). 


The combination of teaching the traditional subjects of science, math, reading, and social 
studies, PLTW STEM units, in conjunction with the attributes described by Morrison (2016) 
and aspects of the PBL learning cycle proposed by Bybee (1997), create the integrated STEM 


model utilized in this study. 
Isolation versus Integration 


Dugger (2010) and Sanders (2009) explained many ways of teaching STEM in kindergarten 
through 12" grades. One method is to instruct STEM individually, teaching science, 
technology, engineering, and mathematics separately in “silos” as individual subjects 
(Dugger, 2010, p. 4), also known as teaching STEM in isolation. Stohlman et al. (2011) 
stated mathematics and science are the subjects most often taught in isolation. Another way is 
to teach all four subjects concurrently and interwoven together in an interdisciplinary method, 
called teaching STEM in integration (Dugger, 2010, p. 4). There are also ways discussed by 
Dugger (2010) that range in between teaching in isolation and integration, such as teaching 
STEM with an emphasis on certain subjects over others, like SteM which is the integration of 
technology and engineering in a primarily science and mathematics classroom. Dugger 
(2010) and Sanders (2009) noted more research needs to be conducted on the success and 
effectiveness of each of these models. Bhattacharya et al. (2015) performed and analyzed 
fully integrated STEM lessons and reported that it is very difficult for science teachers to 
design and facilitate engineering concepts into STEM lessons. Berland (2013), Honey et al. 
(2014), and McGinnis (2017) added full STEM integration, although powerful, has many 
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challenges and more research in science and engineering education needs to be conducted. 
Locke (2009) created proposed models for streamlined and optimized STEM programming 
for kindergarten through 12" grades, containing an engineering focus. Locke’s (2009) 
visionary models were created years before the influx of the integrated and interdisciplinary 


STEM movement and are used as a guide for STEM programming frameworks. 
Some STEM for All or All STEM for Some 


Atkinson (2012) discussed another dichotomy that exists in how STEM education is provided 
to students. Atkinson (2012) reported the majority of stakeholders believe all students should 
be exposed to STEM education in varying degrees and coined the “Some STEM for All” or 
“STEM for All” approach. Gonzalez and Kuenzi (2012) reported analysts argue that this is 
not successful. The opposite method argued by Atkinson (2012) to be more effective, 
particularly at the high school level, is providing intensive STEM education to a limited 
number of students, based primarily on student interest, and is called the “All STEM for 
Some” approach. The dichotomy of STEM-integration delivery into schools is a controversial 
topic with more research needed to determine what method is most impactful on students’ 
success and achievement (Atkinson, 2010). In response to Atkinson’s (2010) claims, Elrod et 
al. (2012) stated, 

In the same way that our society is the richer for developing a citizenry that appreciates 

good music whether or not each person develops into a professional jazz musician, our 


society is richer for cultivating the seedbed of STEM literacy (p. 13). 


Elrod et al. argued that Atkinson’s (2010) ideas to concentrate on providing STEM education 
to fewer students showing interest in the discipline reinforces the exclusivity and disparity of 
minorities and underrepresented groups. They concluded that Atkinson’s (2010) model of 
“All STEM for Some” strengthens the STEM pipeline for a select few. However, providing 
all STEM for a few select students often excludes minorities and underrepresented groups. 
Many researchers consider this a major criticism or the “All STEM for Some” approach to 
providing STEM education (Elrod et al, p.11). Further research must be conducted to 
determine the effectiveness of both STEM integration approaches. This study will add to the 


current research on the impact of STEM integration on student achievement. 
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Education Pathways and Future Careers 


There is a reported decline in student interest in STEM fields as students advance from 
middle to high school. Students are noted to view STEM careers as uncreative and socially 
isolating (Kier et al., 2014; Doerschuk et al., 2016; Simon et al., 2015). Kier et al. argued that 
to maintain the momentum evidenced in middle school into the high school years, students 
must identify with a given STEM career. This is accomplished easily with teachers 
instructing about careers and post-graduation options throughout the middle school years 


(Kier et al.). 


Dooley et al. (2017) stated student retention along the STEM pathway is directly correlated 
with math and science grades. They indicated that factors such as gender, socioeconomic 
status, and immigration status play secondary roles. Although the research was conducted on 
Canadian high schools, it is important to note that Dooley et al. determined two key findings 
in their research. First, the biggest predictor of students not preparing for STEM education in 
high school was enrollment in advanced math and science classes. The second predictor was 
students’ reported grades in math and science. The summation of nonacademic factors such 
as gender, family income, and race played a secondary role (Dooley et al.). Dooley et al. 
concluded that individual academic performance is the biggest predictor of retention along 


the STEM pathway and into STEM careers. 


In 2014, the U.S. Census Bureau reported 74% of college graduates in STEM fields were not 
employed in a STEM field (U.S. Census Bureau, 2014). In 2012, the American Committee 
Survey reported engineering, computer science, math, and statistics were the largest group of 
graduates employed in a STEM field with approximately half working in a STEM 
occupation. College science and social science majors had the fewest number of graduates 
employed in STEM occupations with 26% majoring in physical science, 15% in biological 
science, agricultural, or environmental science, 10% in psychology, and 7% in other social 


science degrees. 
Chapter Summary 


This chapter provided a review of the literature defining STEM education, particularly 


integrated STEM education and the connection to PBL initiatives. Although a common 
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definition of STEM education exists among researchers, there is ambiguity in the shared 
definition of integrated STEM education and implementation in the middle school setting. 
This chapter also reviewed the history of STEM education in the United States, as well as 
local considerations in the post steel-producing states of Ohio, Pennsylvania, and West 
Virginia with a focus on funding and implications for educational leaders. The historical 
impact of STEM programming on student achievement, particularly among underrepresented 
groups, such as women and minorities was discussed. Curriculum and_ standards 
encompassing current STEM education were discussed with the two main dichotomies of 
STEM implementation: isolation versus integration and “Some STEM for All” and “All 
STEM for Some” explained. The need for critical thinkers and technological literacy among 
the future workforce has facilitated increases in STEM education funding making this 
research of determining the impact of middle school STEM programming on student 


performance important. 
The goal of this study is to determine the impact of STEM programming, particularly at the 


middle school level, on students’ achievement to determine the effectiveness of both PLTW 


and PBL in relation to STEM programming. 
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CHAPTER 3: MEASURING STUDENT AND SCHOOL 


EFFECTS ON ACADEMIC ACHIEVEMENT IN A STEM 
PROGRAM 


“If I had an hour to solve a problem and my life depended on it, I would use the first 
fifty-five minutes determining the proper questions to ask.” 


Albert Einstein (date unknown) 


Little research exists on the academic impact of STEM programming on middle school 
students, particularly students participating in integrated STEM education programs. In 
addition, there is ambiguity in defining integrated STEM education among educators and 
researchers. The purpose of this investigation is to define integrated STEM education within 
the context of middle school and determine the impact STEM programming has on student 
academic achievement compared to their grade level peers participating in the general 
education setting. This longitudinal case study analyzed student performance on OST over 
the course of seven years commencing in the 2012-2013 school year and ending the 2018- 


2019 school year. 


This chapter provides an understanding of the methodology used to examine the research 
questions. First, the research question and objectives of the study are defined. Second, the 
research design and description of the quantitative research variables are described. Then, an 
explanation of the setting, sample participants, and staff are discussed with details of each 
component reviewed to provide an overview of the contextual factors in the study. Next, a 
description of the STEM program curriculum and details of implementation are discussed 
chronicling the PLTW curriculum utilized and other STEM-related projects and activities 
encompassing the middle school STEM program researched in this study. The following 
additional project-based learning programs and activities are described: annual weather 


balloon launch, Soap Box Derby racing and competition, and the annual STEM Showcase. 
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Finally, the measures of student academic achievement and performance are described with 


methods of data collection explained. 
Research Question 


The research question addressed in this study is to determine the impact of middle school 
integrated STEM programming on student performance and achievement. Currently, little 
research has been conducted on the impact of STEM programming and instruction on student 
academic performance, particularly performance and achievement. The primary research 


question to be addressed in this research is as follows: 
Does middle school integrated STEM programming positively affect student achievement? 


Additional research questions related to the primary questions include: 

Research Question 1 

Do students participating in an integrated middle school STEM program demonstrate 
differences in academic achievement in ELA, math, and science compared to students 
participating in a traditional general education setting? 

Research Question 2 

Do students participating in the integrated middle school STEM program demonstrate 
differences in achievement due to the interaction effect of gender, race, socioeconomic 
status, or attendance between the STEM program students and general education 


students? 
Objectives of the Study 


The main objectives of this study are as follows: 

1. Define and explain middle school STEM integrated programming. 

2. Determine the interaction effects of demographic factors, such as student race, 
gender, socioeconomic status, and student attendance rates on student achievement 
for students participating in a middle school integrated STEM program compared 
to students participating in the general education setting. To determine interaction 
effects of demographic factors, such as student race, gender, socioeconomic status, 


and student attendance rates on student achievement for students participating in a 
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middle school integrated STEM program compared to students not participating in 
the STEM program. 

3. Determine the effect of middle school STEM integrated programming on student 
achievement. To test student achievement using OST in ELA, mathematics, and 
science for those that participated in middle school STEM programming compared 


to students achievement in the general education setting. 


The key hypothesis of this study is: 
H,: The STEM Program will result in either no change or negative effects on student 
academic achievement which will be tested against the alternative: 


H,: The STEM Program will have positive effects on student academic achievement. 
Research Design 


A quantitative research approach was used to determine the impact of a middle school STEM 
program on student academic performance and achievement. A longitudinal case study was 
utilized to determine student performance over the course of seven years in a middle school 
STEM program. The middle school integrated STEM program researched in this study has 
many contextual factors within the environment which change over the course of many years 
making longitudinal case study research the preferred method for this investigation. Case 
study research is important as case studies allow for a more in-depth level of research 
accounting for the multi-faceted and complex variables that account in real-life situations 
(Crowe et al., 2011). This methodology will explain the various contextual factors that 


impacted the research of this longitudinal case study. 
Quantitative Research Variables 


There were several quantitative research variables in the study anticipated to display varying 
differences among student performance and achievement as reported on OST. The main 
research variable was student participation in an integrated middle school STEM program. 
Another quantitative variable was grade level from sixth to eighth grade. In addition, student 
subgroups, such as socioeconomic status determined by students who receive free and 
reduced lunch, student race, and minority status were determined and included in the study. 


Students with disabilities, migrants, and English language (EL) learners were not part of the 
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study due to the small number of students in those categories being statistically insignificant. 
Gender differences were also reviewed to determine if there was a significant discrepancy in 
performance and achievement between male and female students. This longitudinal study 
analyzed student state assessment data from historically underrepresented populations, such 
as minorities, young women, and economically disadvantaged students participating in this 
STEM program to determine relative achievement gaps and to determine the interaction 
effects of demographic student factors and the school factors, such as STEM program 
participation on academic achievement. The purpose of this study was to determine if the 
STEM program has a positive effect on student performance and achievement indicating it is 


indeed a pathway to success. 
Context and Participants 


The context described in the study was the setting, environment, and details of the setting that 
made the research case study interesting and important. The participants were the students 
and teachers in the research study who participated in implementing the integrated STEM 


program curriculum and instruction. 
Setting 


The setting for the research investigation was a mid-size urban district located in northeastern 
Ohio. During the research investigation the district was largest in student enrollment 
compared to any other school district in the county with 4,437 students enrolled in grades 
kindergarten through 12" grade (ODE, 2019). The district comprises four buildings servicing 
students at the following grade levels: elementary (kindergarten through second grade, 
intermediate (third through fifth grade), middle (sixth through eighth grade), and high (ninth 
through 12" grade). Both the percent of students in the district economically disadvantaged 
and the minority enrollment have remained steady over the seven years of this study. In the 
beginning of the 2012-2013 academic year, the percent of students deemed economically 
disadvantaged, as defined by the number of students receiving a free or reduced lunch, was 
51%. The racial composition of the district was 81% White (non-Hispanic), 10% Black (non- 
Hispanic), 3% Multiracial, 3% Hispanic, and less than 1% Asian or Pacific Islander. By 
2019, the number of students identified as economically disadvantaged had remained steady 


at 51%, with a student enrollment distribution similar to the beginning of the study of 75% 
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White (non-Hispanic), 13% Black (non-Hispanic), 6% Multiracial, 6% Hispanic, and less 
than 1% Asian or Pacific Islander (ODE, 2019). 


The school district of interest in this research study is known for being not only the largest 
district in the county but also one of the most progressive. In this capacity, the district offers 
many alternative educational programs for students. Along with a STEM program which 
launched in 2012, the district also began an online alternative educational option for students 
specializing in online education by providing a blended learning environment and distance 
education for students in grades kindergarten through 12™ grade. Another alternative 
educational program, the HOPE Academy, “Helping Others Pursue Excellence”, was created 
to service at-risk students who had a history of poor grades, truancy, disruptive behavior, or 
other behaviors which have led to temporary or permanent withdrawal from school. Both of 
these alternative programs offer a blended learning model for providing educational services 


to students. 
The demographics of the middle school in the study are similar to the demographics of the 
district. According to the 2018-2019 local report card, the middle school had 1,061 students 


enrolled with the following demographic breakdown by subgroup displayed in Table 1. 


Table 1. Middle School Student Demographics (n = 1,061 students) 


Demographics Enrollment # Percentage % 
Am. Indian/ Alaskan Native NC NC 
Asian or Pacific Islander NC NC 
Black, Non-Hispanic 136 12.8% 
Hispanic 66 6.2% 
Multiracial 64 6% 
White, Non-Hispanic 784 73.9% 
Students with Disabilities 191 18% 
Economic Disadvantage 557 52.5% 
English Learner NC NC 
Migrant NC NC 


Note. The data were compiled by the ODE (2019). 
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Sample Participants 


The middle school of interest in the study was the first school in the county to launch a self- 
contained and integrated STEM program commencing the 2012-2013 school year with 25 
seventh-grade students the first year and an addition of an eighth grade class the subsequent 
year. The population studied in this research includes all students in grades six through eight 
from the 2012-2013 school year until the 2018-2019 school year spanning seven years. The 
2019-2020 school year was omitted from the research because there was no state assessment 
data due to the suspension of testing from the coronavirus school closure. Therefore, the 
longitudinal data are from participants at the middle school in both the general education 
classroom and STEM Program over the course of seven years. Student were self-selected for 
participation in the STEM program with interested students applying for participation. In the 
2018-2019 school year, the sixth grade was added to the program with 26 students. Table 2 
shows the STEM program student enrollment by grade level for each school year 
disaggregated by gender. The disaggregation by gender is displayed to show the number of 


male and female students in each grade level of the STEM program for each school year. 


Table 2. STEM Program Student Enrollment by Grade Level for Each School Year 
Disaggregated by Gender 


School Year Grade Level Total Students Male Female 


2012-2013 7th 25 18 7 
2013-2014 7th 25 18 7 
8th 25 18 7 
2014-2015 7th 23 17 8 
8th 25 18 7 
2015-2016 7th 25 15 10 
8th 23 18 7 
2016-2017 7th 26 16 10 
8th 26 13 13 
2017-2018 7th 27 14 13 
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School Year Grade Level Total Students Male Female 


8th 25 14 10 
2018-2019 6th 26 19 fi 
7th 26 12 14 
8th 28 16 12 


Note. No data were collected for the 2019-2020 school year. 


Staff 


The staff at the middle school in the study were all licensed teachers through the ODE for 
their prospective grade level. According to the 2018-2019 school year, there were 56 general 
education, 14 special education, two fine arts, two music, three physical education, and three 
technology/STEM teachers. Among the entire staff, 100% hold at least a bachelor’s degree 
and 81.6% hold at least a master’s degree (ODE, 2019). Content teachers in grades six 
through eight, in the traditional general education setting, instruct in one of the following 
content areas: mathematics, ELA, social studies, and science. Beginning in the 2013-2014 
school year, all students participating in the general education setting received two elective 
courses each nine weeks. The elective course offerings for both sixth and seventh grades 
included art, physical education, technology, and a STEM module. Students also had the 
options of participation in band or choir for the school year making it their second elective 
course. For the 2013-2014 school year, the sixth-grade PLTW STEM module taught was 
Design & Modeling for sixth grade and Medical Detectives for seventh grade, respectively. 
For the 2014-2015 school year through the 2018-2019 school year, sixth grade students 
received instruction in Design & Modeling and seventh grade received instruction in Energy 
& the Environment. No PLTW STEM modules were offered to general education students for 
the 2012-2013 school year. Eighth grade students received a semester of art and a semester of 
technology with the technology class counting as one quarter credit of high school 
technology credits required for high school graduation. Therefore, eighth grade students did 


not receive PLTW modules in their elective coursework. 
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STEM Program Implementation 


The middle school that was the focus in this study offered STEM programming to traditional 
students using PLTW curriculum taught in isolation. Students in sixth grade received nine 
weeks of instruction in Design & Modeling and seventh grade students received nine weeks 
of Energy and the Environment from technology/STEM teachers. Students participating in 
the STEM Program received full integration of STEM programming, receiving several 
PLTW modules offered in nine weeks courses depending upon grade level. These PLTW 
modules were integrated into core content courses and taught as a full STEM integration 
model. All sixth and seventh grade students participating in the middle school STEM 
program received the PLTW Gateway curriculum modules shown in Table 3. This 
curriculum was taught by two core content teachers each instructing both Math and Science 


and Social Studies/Language Arts, respectively. 


Table 3. STEM Program Modules Taught by School Year and Grade Level 


School Year Grade Level PLTW Gateway Modules Taught 


2012-2013 7th Design & Modeling 
Energy & the Environment 


Automation & Robotics 


2013-2014 7th Design & Modeling 
Energy & the Environment 


Automation & Robotics Part 1 


8th Automation & Robotics Part 2 
Magic of Electrons 
Flight & Space 


2014-2015 through 2017-2018 7th Design & Modeling 
Energy & the Environment 


Automation & Robotics Part 1 


8th Automation & Robotics Part 2 


Magic of Electrons 
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School Year Grade Level PLTW Gateway Modules Taught 
Flight & Space 
Green Architecture 

2018-2019 6th Design & Modeling 


Medical Detectives 


Automation & Robotics Part 1 


7th Design & Modeling 
Medical Detectives 


Automation & Robotics Part 1 


8th Automation & Robotics Part 2 
Magic of Electrons 
Flight & Space 
Green Architecture 


Computer Apps for Inventors & Creators 


Note. Automation & Robotics was split into two courses, Part 1 and 2, due to the 


extensiveness of the course. 


The integrated STEM Program PLTW Gateway modules illustrated in Table 3 show the 
modules taught. These nine-week modules were taught in conjunction with the content area 
courses of science, mathematics, ELA, and social studies in an integrated learning 
environment using PBL. In addition to the described PLTW modules, students participating 
in the STEM program engaged in two comprehensive and interdisciplinary projects annually: 
the building and racing of Soap Box Derby cars and the launching and retrieving of a weather 


balloon. 
Soap Box Derby 
The All-American Soap Box Derby (AASBD) is a gravity-racing program for youths 


between the ages of seven and 20 years old. The AASBD is part of the International Soap 
Box Derby which was founded in 1934 in Akron, Ohio. The AASBD has their own 
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educational racing program designed for schools called the Gravity Racing Challenge (GRC) 
STEM Team Competition. Every year students participating in the STEM program compete 
in the GRC STEM Team Competition by building GRC cars and racing in May at Derby 
Downs in Akron, Ohio. Seventh grade students build three stock cars (designed for children 
between seven and 13 years old) and eighth grade students build three superstock cars 
(designed for students between nine and 18 years old), respectively (Soap Box Derby, 2020). 

Along with the instructional component involved with building the Soap Box Derby cars 
there was an AASBD sponsored and approved curriculum. The curriculum for the GRC 
STEM Team Competition coined the “Masters of Gravity” followed the learning cycle 
proposed by Bybee (1997) focusing on the five pillars of the learning cycle; engagement, 
exploration, explanation, elaboration, and evaluation. The curriculum consisted of eight 
lessons, called programs, of varying lengths designed to encompass all subjects of science, 
technology, engineering, and mathematics. The programs were taught in conjunction while 
building the cars and commenced in the following order: Collection and Analysis, Ratio and 
Proportion, Geometry, Simple Machines, Gravity, Energy, Friction, and Speed. The entire 
“Masters of Gravity” curriculum took approximately two months to complete. The program 
begins in March and resolves with the GRC STEM Team Competition races in May of each 
school year (Masters of Gravity, n.d). In addition to the building of the Soap Box Derby cars 
and implementing the curriculum programs ,students participated in the optional competitions 
associated with the GRC STEM Team Competition of the photography contest, infomercial 
creation, and press release design. The participation of the STEM program students in the 


GRC STEM Team Competition was funded by private sponsors and local donations. 
Weather Balloon Launch 


Every year the eighth grade STEM program students launch a weather balloon as a 
supplemental PBL project complementing the PLTW Flight and Space module. Students 
were responsible for researching and organizing the weather balloon launch which consists of 
a 1200 gram balloon filled with helium and carrying a payload containing two GoPro 
cameras and an onboard flight computer with sensors to collect data such as temperature, 
pressure, and humidity. A select group of eighth-grade STEM students tracked and chased 
the balloon until it was retrieved and the data were then analyzed in the classroom using 


concepts learned in the PLTW Flight and Space module. This is an example of a PBL project 
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encompassing Bybee’s (1997) five aspects of the learning cycle used in many STEM 


programs and activities today. 
Instrumentation 


Student achievement was measured using OST data as instruments and reported by the ODE. 
Student demographic data were collected from the Education Management Information 
System (EMIS) database of the school district which is also reported to the ODE. Student 
demographic information consisted of student race, defined as: American Indian or Alaskan 
Native, Asian or Pacific Islander, Multiracial, Hispanic, Black (non-Hispanic), and White 
(non-Hispanic). Both race and socioeconomic status were self-reported by parents and 
guardians to the school district. Race was self-reported by parents and guardians with no 
required documentation. Socioeconomic status was derived from free- and reduced-lunch 
status reported by family group W-2 forms and was considered a valid metric for assessment 
of socioeconomic status. This study did not examine ELLs and migrants due to the low 


number of students in each population studied. 


The student measure of academic achievement was determined using annual OST scores 
taken each spring by students in grades three through eight for all students in the state of 
Ohio. Testing is mandatory in grades three and above with particular tests by subject required 
at the high school level with an appropriate score required for graduation. Students take 
English Language Arts (ELA) and mathematics OST every year. Science testing occurs for 
students in their fifth- and eighth-grade year. 


OST scores, grades, and other indicators of academic success were retrieved and reported. 
OST scores were obtained from various vendors in different years. These included: The Ohio 
Achievement Assessments (OAA), Partnership for Assessment of Readiness for College and 
Careers (PARCC), and the American Institute of Research (AIR). Table 4 displays the OST 
test name disaggregated by school year and content. There was suspension of state testing for 
the 2019-2020 school year due to the coronavirus crisis which caused school closure and 
suspended state testing. Therefore, no student test data were collected for the 2019-2020 


academic school year. 
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Table 4. Ohio State Test Name by School Year and Test Subject 


School Year Test Subjects Test Name 


2012-2013 through 2013-2014 English Language Arts OAA 


Mathematics 
Science 
2014-2015 English Language Arts PARCC 
Mathematics 
Science AIR 


2015-2016 through 2018-2019 English Language Arts AIR 
Mathematics 


Science 


Scale and Scoring System of Instrument 


Over the course of seven years of this study, the ODE utilized three different testing vendors 
for the administration of state testing: OAA, PARCC, and AIR. These state assessments were 
norm-referenced and mandatory for all students attending schools in the state of Ohio and 
approved as a method of providing standardized education by the NCLB Act of 2001. OAA 
was administered for the 2012-2013 and 2013-2014 school years for both reading and math. 


The Rasch model (a single parameter logistic model) was used for the conversion of raw 
scores to scaled scores. This accounts for the levels of difficulty and abilities of students 
taking the assessment. The Rasch model determines estimates of the level of difficulty of test 
questions and the ability of each student on a linear scale to determine the probability of 
getting each question correct and adjusted scores accordingly. Table 5 reveals the scaled 


scores for each performance level on the OAA. 


The five performance levels in increasing order are as follows: limited, basic, proficiency, 
accelerated, and advanced. For the purposes of data reporting, performance levels are 
numbered from one to five and are coded as followed: limited = 1, basic = 2, proficient = 3, 


accelerated = 4, and advanced =5. All assessments used the same coding system for each 
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performance level. Proficiency or passage was determined to be a scaled score of 400. It is 


important to note that scaled scores are not comparable between grade levels and test 


subjects. Reliability of all OAA assessments across subjects and test years ranged from 0.87- 


0.90 using Cronbach’s alpha and the standard error of measurement (SEM) ranged from 


10.24-13.03, respectively (ODE, 2014). 


Table 5. 2012-2013 and 2013-2014 Cut Scaled Score Points for Basic, Proficient, 


Accelerated, and Advanced Standards, OAA Administration 


Grade = Subject Limited Basic Proficient Accelerated Advanced 
Reading —a09 319 400 432 452 

: Math <378 378 400 436 458 
Reading <378 378 400 428 451 

; Math <379 379 400 432 459 
Science <365 365 400 427 445 


In the 2014-2015 school year, the ODE used PARCC as the mandated Ohio state assessment. 


Table 6 illustrates the scaled scores for each performance level with proficiency determined 


with a scaled score of 725. Similar to OAA, scaled scores are not comparable between grade 


levels and test subjects (ODE, 2015). 


Table 6. 2014-2015 Scale Score Ranges for All Subjects and Performance Levels, PARCC 


Administration 
Grade Subject Limited Basic Proficient Accelerated Advanced 
7 ELA 650-699 700-724 725-749 750-784 785-850 
Math 650-699 700-724 725-749 750-785 786-850 
8 ELA 650-699 700-724 725-749 750-793 794-850 
Math 650-699 700-724 725-749 750-800 801-850 
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Beginning in the 2015-2016 school year and continuing through the 2018-2019 the ODE 
administered the AIR assessments. Table 7 illustrates the cut scaled scores for each 
performance level similar to the OAA and PARCC assessments. Passing is defined as 
achieving proficiency level or getting at least a 700 scaled score on the assessment. Similar to 
the state assessments mandated before AIR, scaled scores are not comparable between grade 
levels and test subjects. In addition, students receiving a DNA or INV in data reports either 


“did not attempt” or the data was “invalidated”, respectively (ODE, 2019). 


Table 7. 2015-2016 through 2018-2019 Scale Score Ranges, AIR Administration 


Grade Subject Limited Basic Proficient Accelerated Advanced 


6 ELA 555-667 668-699 700-724 725-750 751-851 
Math 616-681 682-699 700-724 725-743 744-790 
i ELA 568-669 670-699 700-724 725-748 749-833 
Math 605-683 684-699 700-724 725-754 755-806 
ELA 586-681 682-699 700-724 725-743 744-805 
: Math 633-689 690-699 700-724 725-743 744-774 


Science 575-673 674-699 = 700-724 725-765 766-868 


Reliability and Validity of the Instrument 


The reliability and validity of Ohio state assessment data are reported annually through the 
ODE. Reliability of all OAA assessments across subjects and test years ranged from 0.87- 
0.90 using Cronbach’s alpha and the standard error of measurement (SEM) ranged from 
10.24-13.03, respectively (ODE, 2014). Reliability of AIR assessments across reading, math, 
and science tests for the 2015-2016 through the 2018-2019 school years ranged from 0.90- 
0.94 using Cronbach’s alpha and the SEM ranged from 9.81-15.49 (ODE, 2020). Reliability 
measures from the one year (2014-2015) administration of the PARCC assessment could not 


be found, perhaps because the assessment was only given for one year. The reported 
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Cronbach’s alpha measures for calculating internal consistency are all well above 0.70 


indicating all administered assessments have strong reliability. 


The validity of using OST scores as an instrument for determining achievement is commonly 
used by researchers and is considered one of the stronger methods of defining achievement. 
State assessments are norm-referenced and standardized to ensure alignment with Ohio’s 
Learning Standards for each grade level and subject. The process of creating test questions 
and scoring parameters was completed by assessment committees in several areas: content 
advisory, range finding/ rubric validation, fairness/sensitivity, standard setting, and alignment 
study committees (ODE, 2020). Each committee is made up of licensed Ohio educators and 
higher education members that specialize in these specific areas. In addition, many different 
stakeholders were consulted to ensure Ohio state assessments have content validity and are 
accurately measuring students' knowledge based on the standards of each tested subject and 


grade level (ODE, 2020). 
Data Collection Procedure 


Permission for this research project was granted through the school district’s superintendent 
in accordance with the local school district policy, and also through the Youngstown State 
University (YSU) office of Institutional Research. Institutional Review Board (IRB) approval 
was obtained under exempt status (see Appendix B). After IRB approval, existing and de- 
identifying OST data were sent electronically via email. All individual student demographic 
and OST data were retrieved from the Ohio Department of Education Secure Data Center 
online using the appropriate permissions and removing student names and student state 
identification (SSID) numbers prior to being sent electronically to the researcher. The 
electronic data encompassed individual student demographic and OST data from school year 
2010-2011 through 2018-2019 taken over the course of nine years. Students were linked with 
randomly assigned identification numbers with data organized into spreadsheet files 


compatible with IBM SPSS for data analysis. 
Data Analysis Methods 


The method of data analysis used in the study was multilevel modeling. Multilevel modeling 


is used in many fields of study particularly in education, social work, health, business sectors, 
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and the social sciences (Woltman et al., 2012). This type of modeling is known by several 
names, such as hierarchical linear-, mixed level-, mixed effects-, random effects-, random 
coefficient (regressions), and (complex) covariance components- modeling (Raudenbush & 
Bryk, 2002). Multilevel modeling and HLM are complex forms of ordinary least squares 
(OLS) regression and are used when predictor variables are at different hierarchical levels to 
determine the variance within the outcome variables. HLM is primarily used for creating 
statistical models of variables that depend on more than one level, or nested data. HLM 
simultaneously determines relationships within and among hierarchical levels within data sets 
thereby making it an effective method of calculating variance among variables at varying 
levels than other statistical analysis techniques. HLM is becoming an increasingly popular 
method of advanced statistical analysis due to advancement in statistical theory and statistical 


modeling programs (Woltman et al.). 


The goal of this study was to determine the academic achievement of individual students as a 
function of different levels within nested data sets. This aligns with the main objective of this 
study to determine the effect of middle school STEM integrated programming on student 
achievement using OST scores as a measure of achievement. The related research question 
contains two hierarchical levels: what grade level and classroom level, and student-related 
factors affect student achievement? Table 8 displays the three hierarchical levels defining 
their category and factors, also known as variables at each particular level. The variable name 
used during SPSS and HLM analysis is listed also. HLM was used to analyze OST data for 
students in grades six through eight to determine the effects of student achievement, the 


outcome variable, as a function of varying hierarchical levels. 


The higher level, (level-2) is the school-related variables pertaining to grade level, which are 
grades six, seven, and eight depending on school year. The other level-2, school-related 
variables are whether students participated in a STEM program or a general educational 
setting. Level-1 variables are located within level-2 groups. The level-1, student-related 
variables in this research study consist of OST scores, gender, race, and socioeconomic 
status. These level-1 variables are located within groups inside Level-2 and together are 
influenced by level-2 variables. To summarize, students (level-1) are encapsulated in 
classrooms (level-2) that are nested inside of schools (see Table 8). The OST score, which is 
the outcome variable, is assessed at level-1. In HLM, the targeted variable of interest and the 


outcome variable are found at the lowest hierarchical level (Woltman et al., 2012). 
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Table 8. Factors at Each Hierarchical Level that Affect Students’ Achievement with 


Variables’ Names 


Hierarchical Category Variables Variable 
Level Name 
Level-2 School Level Participation ina STEM program STEM 
Participation in the general education 
setting 
Grade 6 GRADE6 
Grade 7 GRADE7 
Grade 8 GRADE8 
Level-1 Student OST scores OST 
Level Gender GENDER 
Race RACE 
Socioeconomic Status SES 
Attendance ATTEND 


HLM can accommodate more than one outcome variable in one analysis. In addition, the 
outcome variables can be discrete or continuous (Raudenbush & Bryk, 2002). For the 
purposes of this research study, the primary outcome variable was OST scores, however, the 
other level-1 variables consisting of gender, race, socioeconomic status, and student 
attendance rate were analyzed using HLM to determine variance among variables. The 
coding of discrete variables were given the following number system: GENDER; female= 1 
and male=0, RACE= Am. Indian/ Alaskan Native=1, Asian=2, Black=3, Hispanic=4, 
Multiracial=5, Puerto Rican=6, White (Non-Hispanic)=7 and SES; yes=1 and no=0. Student 
attendance rate was coded as percent attendance as a decimal rate. For example, a student 
present every day of school, 186 days out of 186 school days was coded as ATTEND=1. A 
student attending exactly half of the days was coded as ATTEND=0.5, and a student enrolled 
who never attended was coded as ATTEND=zero. This ensured student attendance did not 
skew data for those coming a portion of the school year. This also ensured students with poor 


attendance were accounted for correctly within the data distribution. 
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The use of HLM is the preferred method of analyzing multilevel datasets as it accounts for 
the communal variance fundamental to hierarchical datasets. Basic linear regression methods, 
disaggregation, and aggregation, which did not consider hierarchical data and their shared 
variances, were used by researchers prior to the use of HLM. Although both of these outdated 
methods made analyzing hierarchical data possible they created other problems. These 
included the wrong variances assigned between variables, data dependencies, and an 


increased probability of a Type I error (Woltman et al., 2012). 


Disaggregation of data ignores the differences in hierarchical structure among the data 
treating all relationships between variables at hierarchical level-1, or the individual level. 
This method of data analysis disregards between-group differences in variances. Table 9 
shows a hypothetical dataset if disaggregation was to be used in this study. The following 
dummy variables are defined as: gender; I=female and O=male, STEM Program 
Participation; 1=yes and 0=no. The upper level variables of level-2 (school) are treated as 


level-1 (student). 


All students in a STEM program would be attached to similar classroom-related OST scores, 
and all participants in similar grade levels would be attached to similar grade-level OST 
scores, thus bringing all upper level hierarchical variables down to level-1. When this occurs, 
data dependencies are not corrected, statistical methods are established only from the sample 
size at level-1, and the probability of measuring variances falsely increases. In addition, the 
independence assumption required for simple linear regression is violated. All of these 
factors and the disregard for variance differences between groups using the disaggregation 
method is the reason HLM is the preferred method of statistical data analysis for this research 


study (Woltman et al., 2012). 


The other method used to analyze multilevel data prior to the widespread use of HLM is 
aggregation. Aggregation is a simple linear regression method that disregards lower level 
individual differences, instead of ignoring upper level differences, such as with 
disaggregation. Level-1 variables are treated at a higher level making variability among 
individuals disappear and all students are treated as homogenous entities (Gill, 2003; 


Woltman et al., 2012). 
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Table 9. Sample Dataset Using the Disaggregation Method, with Level-2 and Level-3 
Variables Excluded from the Data 


Student ID SchoolID Grade ID OST Score STEM Program Gender 


(Level-1) (Level-2) (Level-3) (Level-1) Participation (Level-1) 
(Level-2) 
1 1 6 724 1 1 
2 1 6 734 1 0 
3 1 6 701 0 1 
4 1 7 688 1 0 
5 1 7 723 1 1 
6 2 7 714 0 0 
7 2 7 678 0 0 
8 2 8 704 1 0 
9 2 8 697 1 1 
10 2 8 699 0 1 


Table 10 shows a sample dataset of the aggregation method used in this study with all level-1 
variables treated as higher hierarchical levels. The aggregation method makes the mean 
classroom or grade OST score become the targeted outcome variable of focus, rather than 
individual student achievement indicative by OST score. Raudenbush and Byrk (1992) 
reported approximately 80%-90% of deviation among individual differences is lost when the 
aggregation method is used. HLM is often chosen instead of aggregation in reference to 
hierarchical information due to its successful separation of group and individual effects on 


the variable of interest (Woltman et al., 2012). 
The hypothesis in this research study is students participating in an integrated middle school 


STEM program will demonstrate more academic achievement in math and science than 


students participating in a traditional general education setting. The null hypothesis is the 
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absence of any effect. In addition, the determination of any effect of other level-1 predictor 
variables described earlier: gender, student race, socioeconomic status, and attendance will be 


conducted using HLM. 


Table 10. Sample Dataset Using the Aggregation Method, with Level-1 Variables Excluded 
from the Data 


Classroom ID Classroom OST Score STEM Participation 


(Level-2) (Level-2) (Level-2) 
1 724 1 
2 734 0 
3 701 1 
4 688 0 
5 723 1 


In HLM, the mathematical theory and equations help conceptualize the concept of lower- 
level units representing individual students and higher-level units representing classrooms or 
whole grade levels. The complexity of HLM calculations increases exponentially with every 
increase in hierarchical level (Raudenbush & Bryk, 2002, Woltman et al., 2012). Therefore, 
the mathematical equations were more simply described based on a two-level hierarchical 
model. Of note, a two-level model was utilized in this research study instead of a three-level 
model due to these reasons. The simple linear regression created for each individual student i: 
Yij = Boj + Bij Xy + ry (1) 

where: 

Yj; = dependent variable measured for the ith level-1 unit nested within the jth level-2 
unit, 

Xj; = value on the level-1 predictor, 

80; = intercept for the jth level-2 unit, 

8; = regression coefficient associated with the X, for the jth level-2 unit, and 


rj = random error associated with the ith level-1 unit nested within the jth level-2 unit. 


(Woltman et al., 2012) 
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In context to the research problem the variables can be redefined as follows: 
Yj; = OST scores for student 7 in classroom j 
Xj = Participation in a STEM Program for student 7 in classroom j 
80, = OST scores for student i in classroom 7 who did not participate in a STEM 
program 
§\; = regression coefficient associated with participation in a STEM program for the j 
classroom 


rj, = random error associated with student i 


One important assumption of HLM is that any level-1 errors (7) are normally distributed 
with a mean of zero and a variance equal to o” (Woltman et al., 2012). For a level-2 model, 
the level-1 regression coefficients (Bo; + Bij) are used as outcome variables and are connected 
to the level-2 predictors. The mathematical equation at level-2 becomes increasingly complex 
and supports the use of computerized statistical modeling programs, such as IBM SPSS 


software. 


In a level-2 model, the level-1 regression coefficients of 8o;, OST scores for student 7 in 
classroom j who did not participate in a STEM program, and f\;, the regressions coefficient 
associated with participation in a STEM program for the 7 classroom are used as outcome 
variables and are connected to each of the level-2 (classroom) predictors. For this reason, 
level-2 models are also called between-unit models as they describe the variability among 
many groups (Gill, 2003; Woltman et al., 2012). Equations and 2 and 3 conceptualize the 


case of a single level-2 predictor: 


Bo = Yoo + YorG; + Ug (2) 


Bi =Yi0 t¥G, + Uy (3) 


where: 
Bo; = intercept of the jth level-2 unit; 
Bi; = slope for the jth level-2 unit; 
G; = value on the level-2 predictor; 
Yoo = overall mean intercept adjusted for G; 
Yor = regression coefficient associated with G relative to level-1 intercept; 


Y11 = regression coefficient associated with G relative to level-1 slope; 
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Up; = random effects of the jth level-2 unit adjusted for G on the intercept; 
U;; = random effects of the jth level-2 unit adjusted for G on the slope. 


In context to the research problem, the variables can be redefined as follows when 
considering any level-1 predictor variable, in this example socioeconomic status: 

Bo; = intercept of the jth classroom; 

B7; = slope for the jth classroom; 

G; = participation in a STEM or general education program; 

Yoo = overall mean intercept adjusted for SES; 

¥ 10 = overall mean intercept adjusted for SES; 

Yor = regression coefficient associated with SES relative to level-2 intercept; 

Y11 = regression coefficient associated with SES relative to level-2 slope; 

Uo; = random effects of the jth classroom adjusted for SES on the intercept; 


U,; = random effects of the jth classroom adjusted for SES on the slope. 


The example given above illustrates how to determine the effect of variances in between 
groups among hierarchical data. Socioeconomic status (SES) is one of the level-1 (student) 
predictors that was analyzed using HLM to determine its effect on students OST scores as a 
dependent function of a level-2 (classroom) predictor of either STEM or general education 
programming. It is important to note that the level-2 (classroom) model brings two new terms 
(Uo; and U;;) that Woltamn et al. (2012) identified as both unique to HLM. This allows for 
the model to determine an estimation of error that normal linear regression cannot determine. 
The covariance between By, the intercept of the jth classroom, and B;;, the slope for the jth 
classroom is equal to the covariance between Up, the random effects of the jth classroom 
adjusted for socioeconomic status and U;;, the random effects of the jth classroom adjusted 
for socioeconomic status on the slope. The assumptions of the level-2 (classroom) models are 


as follows (Raudenbush & Bryk, 2002, Woltman et al., 2012): 


E(Ugy) = 0; E(U;;) =0 
E(Bo) = Yoo: Eby) = Yor (4) 


var(Baj) = var (Uj) = Too 3 var(B i) = var(Uy) = T11 
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cov(By;, Bo) = cov(Ug, U1) = To: 


cov(Uy, 11) = cov(Uy, r1j) = 0 


A combined model of equations 2 and 3 can be created to create equation 5. This combined 
model contains both level-1 (student) and level-2 (classroom) predictors (Xj or 
socioeconomic status and G; or participation in a STEM or general education program) and a 
term going across levels (G,X; participation in a STEM or general education program 
multiplied by socioeconomic status). The error is represented by U;;Xj + Uo + ry. Equation 5 
is called a mixed model as it contains both random and fixed effects which is unique to HLM 


(Woltman et al., 2012). 
Yij = Yoo + Y10Xy + YorG; + YG Xy +UyXy + Ug t ri (5) 


The combined model in equation 5 is similar to a normal regressions but adds two new terms, 
U;; and Ug, which introduces an error estimation that is not a part of normal regression. 
Equation 5 demonstrates a dependency between level-1 (student) clustered within level-2 
(classroom). In addition, U;; and Uy, may possess different values between level-2 units 


creating heterogeneous variances of the error terms. 


According to Woltman et al. (2012), there are five conditions that must be met in order to use 
HLM appropriately and effectively. Conditions two and three must be met prior to conditions 
four and five. The following conditions apply to this research study: 
Condition 1: There is systematic within- and between- group variance in OST scores. 
Conditions 2 and 3: There is significant variance in the level-1 (student) intercept and 
slope. 
Condition 4: The variance in the level-1 (student) intercept is predicted by participation 
in a STEM or general education program. 
Condition 5: The variance in the level-1 (slope) is predicted by participation in a STEM 


or general education program. 


IBM SPSS software originally stood for Statistical Package for Social Sciences but now is 


only known now by the acronym. This widely-used statistical analysis package was used to 
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perform HLM statistical calculations in this research study. The IBM SPSS data software was 


used due to its ability for prediction among nested groups using cluster analysis of HLM. 


There are several advantages to using the multilinear modeling technique called HLM. First, 
the ability of HLM to simultaneously determine relationships between groups makes it very 
efficient for determining variances among different leveled groups. Second, HLM does not 
violate many of the statistical assumptions necessary in using older statistical techniques such 
as disaggregation and aggregation. This type of multilevel modeling is more accepting of 
violations in observation independence, homogeneity, and sphericity. HLM has little effect 
on standard errors, effect size, and variances. Finally, HLM is an effective method of 
determining the differences in variances between nested data as discussed previously (Gill, 


2003; Woltman et al., 2012). 
Internal and External Validity of the Research Design 


Students within this research study were situated into schools which were nested into 
classrooms with no manipulation of independent variables. The large sample size at level-3 
(approximately 350 students) and the longitudinal nature of seven years increased the study’s 


external validity. 
Analytical Limitations 


Although there are many advantages to using HLM and other multilevel, statistical modeling 
techniques, Dedrick et al. (2009) discussed several limitations and concerns to using HLM 
for data analysis that are applicable to this research study. The limitations fall into one of four 
categories: model development, hypothesis testing, data considerations, and estimation 
processing. Model development and specification issues are troublesome when determining 
and selecting predictor variables. The regression equations can become extremely 
complicated, especially when it is not feasible for predictor variables to have zero points. A 
second limitation related to using HLM for hypothesis testing and statistical inference occurs 
when sample sizes or variances parameters are small. Due to small sample size, degrees of 
freedom may need adjustment when there is a violation of normality. Dedrick et al. 
commented that there are two methods of making inferences to overcome this problem. One 


method is to estimate the level-1 (grade level) coefficient separately from level-2 (classroom 
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level) using OLS, which has its own limitations. Also, the researcher can use empirical Bayes 
estimates, which consider all data but bias estimates. However, Bayes estimates tend to 
generate values more accurate to the parameter values. Dedrick et al. stated that there is, in 
fact, no estimation method which satisfies all conditions. Considerations of sample size and 
normality can assist researchers in determining which estimation method is the most 


appropriate for the research study (Dedrick et al.). 


An additional limitation to using HLM emphasized by Woltman et al. (2012) for traditional 
applications of HLM, is that substantial sample sizes are necessary at each level for sufficient 
power. Recently, researchers have overcome this problem by increasing the number of groups 
instead of increasing the number of observations per group. Groups of less than 50 could 
yield biased approximations of standard errors at the second level (Woltman et al., 2012). For 
the purposes of this research design, the level-2 (grade level tests) had approximately 9 
groups and level-1 (classroom level) contained approximately 3,000-4,000 students, 


depending on the test year. 


There are a few limitations to the data collection and research design not necessarily related 
to the use of HLM. The study population is limited to a single institution versus data 
collection from other similar programs in the state. If other programs were included, the 
significance of the individual program teachers and instructors would be diminished. In this 
dataset, race was self-reported by the parent or guardian, leading to a degree of potential 
inaccuracy. The transition from PARCC to AIR assessments for mathematics and ELA can 
create reliability issues as two different assessments were used to measure student 
performance and achievement. Other factors seemingly unrelated to the STEM program may 
have effects on student performance and achievement. It is hypothesized that teacher 
experience and self-efficacy plays an important role in student achievement and intellectual 
development. Finally, migration of students in and out of the STEM program may be a threat 
that affects both internal validity and reliability. 


Chapter Summary 
The reliability and validity measures of state assessment scores are dependent upon the 


accuracy reported by the makers of the OAA, PARCC, and AIR assessments. Reliability 


measures of state assessments across all years in this study and test subjects were reported to 
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be strong with Cronbach’s alpha values greater than 0.7. SEM values were fairly low, 
depending on performance level scaled score values. Norm-referenced and standardized state 
assessments are considered the gold standard for quantifying student achievement. This is 
why OST scores were used in this study. Demographic data of students were self-reported by 
parents. The school identified gifted students and English learners. Economically 
disadvantaged data were determined via student participation in the free or reduced lunch 


program. The validity of parent-reported measures cannot be controlled. 


The use of HLM for data analysis is the preferred method of analyzing multilevel datasets as 
it accounts for the shared variance inherent within hierarchical data. The purpose of using 
HLM in this study was to determine the impact of STEM programming on achievement when 
students are nested within level-2 (classrooms) which are clustered into level-3 (grade levels). 
The level-1 (student) variables of OST scores, gender, race, and socioeconomic status were 
used to determine the variances of in-between groups. With participation in STEM or general 
education programs being a predictor of the level-1(student) slope, the use of HLM was used 
to determine the level of moderation between STEM programming and predicting for all 
level-1 (student) predictors. In conclusion, the percent of variance in regard to STEM 
programming can be calculated as a moderator in the OST-score and demographics 
relationships, including but not limited to gender. The hypothesis in this research study was 
students participating in an integrated middle school STEM program will demonstrate more 
academic achievement in ELA, math, and science than students participating in a traditional 
general education setting when correcting for variances among other level-1 (student) 


demographic variables. 
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CHAPTER 4: FINDINGS FOR STUDENT AND SCHOOL 


EFFECTS ON ACADEMIC ACHIEVEMENT IN A STEM 
PROGRAM 


I 


“We should not teach children the science but give them a taste for them.’ 


Jean Jaques Rosseau (date unknown) 


The goal of this study was to determine the impact of STEM programming, particularly at the 
middle school level, on students’ achievement to determine the effectiveness of both PLTW 
and PBL in relation to STEM programming. Student achievement was measured by the 
scaled score on Ohio State Tests (OST) in math and science over the course of seven years. 
The proposed factors affecting student achievement were STEM participation, gender, race, 
socioeconomic status, and attendance. Table 11 displays the two hierarchical levels of school 
and student level defining their category and factors, also known as variables, at each 
particular level. HLM was used to analyze OST data for students in grades five through eight 
to determine the effect of student achievement, the outcome variable, as a function of varying 


hierarchical levels. 


This chapter displays the variable and sample descriptions with descriptive statistics related 
to each research question. The descriptive statistical information for OST scores by year are 
displayed indicating the mean, standard deviation, minimum score, maximum score, 
skewness, and kurtosis for each subject, grade level, and year. The inter-correlation between 
variables were shown with preliminary bivariate relationships indicated by Pearson 
correlation coefficients. Two HLM models were created addressing the research questions. 
The higher level, (level-2), are the school-related variables pertaining to grade level, which 
are grades 6, 7, and 8th, depending on school year. The other level-2, school-related variables 
represent whether students participated in a STEM program or a general educational setting. 
Level-1 variables are located within level-2 groups. The level-1, student-related variables in 


this research study consist of OST scores, gender, race, and socioeconomic status. These 
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level-1 variables are located within groups inside Level-2 and together are influenced by 
level-2 variables. To summarize, students (level-1) are encapsulated in classrooms (level-2) 
(see Table 11). The OST score, which is the outcome variable, is assessed at level-1. In 
HLM, the targeted variable of interest and the outcome variable are found at the hierarchical 


level at the lowest value (Woltman et al., 2012). 


Table 11. Proposed Factors at Each Hierarchical Level That Affect Students’ Achievement 


Hierarchical Category Variables HLM Variable 
Level Code 
Level-2 School Level Participation ina STEM program STEMMARK 


Participation in the general 

education setting 

Assessment Type ASSMTTYP 
(grade and school year) 

5". and 8"- grades’ science tests | GRADE 


Level-1 Student Level OST scaled score SCALEDSC 
Gender GENDER 
Race RACE 
Socioeconomic Status SES 
Attendance ATTEND 


HLM can accommodate more than one outcome variable in one analysis. HLM is often used 
in place of the Analysis of Covariance (ANCOVA) because HLM can analyze multilevel data 
without homogeneity among regression slopes necessary in ANCOVA. In addition, the 
outcome variables can be discrete or continuous (Raudenbush & Bryk, 2002). For the 
purposes of this research study, the primary outcome variable was OST scores, however, the 
other level-1 variables consisting of gender, race, socioeconomic status, and student 
attendance rate were analyzed using HLM to determine variance among variables. The 
coding of discrete variables provided the following number system: gender: female= 1 and 
male=0, race= Am. Indian/ Alaskan Native=1, Asian= 2, Black=3, Hispanic=4, 
Multiracial=5, Puerto Rican=6, White (Non-Hispanic)=7, and economic disadvantage; yes= | 


and no=0. 
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Variable and Sample Descriptions 


The student level (level-1) variables and the corresponding variable label indicating the 


variable coding are shown in Table 12. 


Table 12. Student Level (Level-1) Variables with the IBM SPSS (v. 26) Label and Variable 


Coding 

Student Level Label Variable Coding 

Variables 

Socioeconomic status SES Economically disadvantaged=1, Not 
economically disadvantaged= 0 

Gender GENDER Female=1, male=0 

Race RACE White (Non-Hispanic)=7, Puerto Rican= 6, 
Multiracial=5, Hispanic=4, Black (Non- 
Hispanic)=3, Asian=2, Alaskan Native/ Am. 
Indian=1 

Attendance ATTEND Between or equal to 0 and 1. Coded as percent 
attendance at a decimal rate. 

OST Score [Year Subject] Ohio State Assessment scaled score for a given 


academic year and subject; Subject is coded as 
math or science; Year is 2013 through 2019; 
5". grade science data 2010 through 2016 


Attendance rate was coded as percent attendance at a decimal rate. This was calculated for 
students by taking the number of present days for the specific school year and dividing by the 
summation of days present, days absent (unexcused) and days absent (excused). The quotient 
was a decimal rate used as the ATTEND code for a given student in a specific school year. 
This ensures accuracy in determining intercorrelation between student attendance and 
achievements. Table 13 displays the school-level variables with the given IBM SPSS (V. 26) 
labels and codes indicating student grade levels of five, seven, and eight. Student 
participation in integrated STEM programming was indicated by 1 and student participation 


in a general education setting was shown by 0. 
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Table 13. School Level (Level-2) Variables with the SPSS Label and Variable Coding 


School Level Variables Label Variable Coding 


Grade Grade Grade 5= 5, Grade 7=7, Grade 8=8 
Participation in a STEM Student participation in a STEM program=1, 
STEM Program Student participation in a general education 


setting and not in a STEM program=0 


The student level (level-1) OST data collected for each academic year beginning in the school 
year 2010-2011 and concluding in 2018-2019 is displayed in Table 14 along with the grade 


level and test subject. 


Table 14. Student Level (Level-1) OST Data Collected by Academic Year, Grade Level, and 
Test Subjects 


Academic Year 5th grade 7th grade 8th grade 


2010-2011 Science 

2011-2012 Science 

2012-2013 Science Math 

2013-2014 Science Math Math Science 
2014-2015 Science Math Math Science 
2015-2016 Science Math Math Science 
2016-2017 Math Math Science 
2017-2018 Math Math Science 
2018-2019 Math Math Science 


Notice that school year 2010-2011 and 2011-2012 OST data were collected for fifth-grade 
students only. The STEM program began in 2012-2013 for students in 7th grade only, 
however fifth-grade science OST data were collected to determine the effect on science 
achievement by comparing fifth-grade science scores to eighth-grade science scores after 
participation in two years of integrated STEM programming. Therefore, all fifth-grade OST 


data were used for control data for comparison upon treatment (two years of integrated 
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STEM programming) and comparison with eighth grade science OST data. Thus, fifth-grade 
science OST data only needed to be collected commencing in the 2010-2011 school year and 


concluding in the 2015-2016 school year. 
Data Screening and Assumptions 


All necessary data were received in two Excel spreadsheet files. The first file contained all 
raw data necessary for the study with each individual participant’s unique student number 
and OST score indicated as a row in the spreadsheet. All other data were located in a column. 
The second file listed the student numbers for STEM participants. There were several 
necessary modifications made to the files in order to create one spreadsheet compatible for 


analysis via SPSS. 


First, the spreadsheet was reorganized with each unique student number as a row and 
individual OST scores, demographic information, and attendance for each year as a column. 
OST scaled score was used in the study and the raw scores and number of test attempts were 
removed from the data spreadsheet. Second, raw attendance information was given in three 
columns; number of days present, number of unexcused days absent, and number of excused 
days absent. To determine the attendance rate as a decimal, a new column was created adding 
excused absences, unexcused absences, and days present to get the total. Then the number of 
days present was divided by the total number of days. This equation proved accurate given 
each school year contained a different number of days in a school year. In addition, the 
equation corrected for students attending the school or district only part of the school year. 
Finally, the student numbers of STEM participants were merged into the spreadsheet 
containing all data as a new column indicating STEM designation (STEM=1) and all other 


participants a zero. 

Missing Data & Outliers 

There was a substantial number of participants missing attendance information for any given 
year. In SPSS, those cells were identified as missing in SPSS using ATTEND=9999 and 


coding “9999” as a missing data point. In addition, only students with OST scores were 


included in the initial data spreadsheet. However, a few students had an OST score for math 
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in a given year but no score indicated for ELA and vice versa. Students with missing scores 


were removed from the study. There were no outliers indicated in the data. 


Descriptive Statistics Related to Each Research Question 


Descriptive statistics for both student and school-level variables are displayed and explained 
below as they are related to each research question. 
Research Question 1: 
Do students participating in an integrated middle school STEM program demonstrate 
differences in academic achievement in math and science compared to students 


participating in a traditional general education setting? 


Descriptive statistics for student OST scores are segregated by STEM participation in Table 
15 through Table 23 beginning in school year 2010-2011 with fifth-grade science OST and 
concluding with 2018-2019 seventh-grade math and seventh- and eighth-grade math and 
science. Tables 15 shows fifth-grade science OST scores for the 2010-2011 school year for 


students participating in general education compared to students in the STEM program. 


Table 15. Descriptive Statistics for 5""-Grade Science for the 2010-2011 School Year (n= 


414) 
Mean SD Min. Max. Skew. Kurt. 
Gen. Ed. 412 26.47 327 527 23 72 
(n= 390) 
STEM 438 23.79 400 490 74 11 
(n= 24) 


Table 16 shows fifth-grade science OST scores for the 2011-2012 school year for students 
participating in general education compared to students in the STEM program. For the 2010- 
2011 school year, the mean OST score was considerably higher for STEM students than 
general education students at 438 and 412, respectively. For both years, the range between 
the minimum and maximum OST scores was much larger for general education students 
(min.= 327, max. 527; min. 314, max. 548) compared to STEM students (min. 400, max. 
490; min. 404; max. 501), respectively. Both the skewness and kurtosis (ranging from .23 to 
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.74 and .11 to .72, respectively) were within acceptable limits indicating the data distribution 


was symmetric and not heavy- or light-tailed. The 2012-2013 school year was the first year 


OST data were collected for both math and science which continued through the 2018-2019 


school year for purposes of this study. 


Table 16. Descriptive Statistics for 5""-Grade Science for the 2011-2012 School Year (n= 


Mean SD 
Gen. Ed. 419 30.98 
(n= 377) 
STEM 447 26.20 
(n= 21) 


398) 


Min. 


314 


404 


Max. 


548 


501 


Skew. Kurt. 
-.013 25 
40 97 


Table 17 displays fifth-grade science, seventh-grade math, and eighth-grade math for the 


2012-2013 school year. 


Table 17. Descriptive Statistics for 5""-Grade Science and 7"-Grade Math for the 2012-2013 


Test Mean 
Grade 5 Gen. Ed. 411 
Science (n= 365) 


STEM 445 
(n= 24) 


Grade 7 Gen. Ed. 422 
Math (n= 424) 


STEM 442 
(n= 28) 


School Year 
SD Min. 
en eye 318 
27.00 390 
31.29 346 
23.49 390 


Max. 


504 


519 


507 


472 


Skew. Kutt. 
Ag .02 
37 1.3 
22 =.52 
-.46 .63 


Similar to previous years, the mean OST score was higher for STEM students with means of 


445 for 5"-grade science and 442 for 7"-grade science compared to 411 and 422, 


respectively. In addition, the range between the minimum and maximum OST scores was 


significantly larger for general education students for all three tests. For example, for fifth- 
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grade science, the range was 186 points (min.= 318, max. 504) compared to STEM students 
with a range of 129 (min. 390, max. 519). Both the skewness and kurtosis (ranging from -.46 
to .37 and -.52 to .63, respectively) were within acceptable limits indicating the data 


distribution was symmetric and not heavy- or light-tailed. 


Table 18 displays fifth-grade science, seventh-grade math, and eighth-grade math and science 


for the 2013-2014 school year. 


Table 18. Descriptive Statistics for 5"_Science, 7"_Grade Math, 8""_Grade Math, and ee 


Test 


Grade 5 


Science 


Grade 7 
Math 


Grade 8 
Math 


Grade 8 


Science 


Gen. Ed. 
(n= 351) 


STEM 
(n= 22) 


Gen. Ed. 
(n= 399) 


STEM 
(n= 25) 


Gen. Ed. 
(n= 424) 


STEM 
(n= 27) 


Gen. Ed. 
(n= 425) 


STEM 
(n= 28) 


Grade Science for the 2013-2014 School Year 


Mean 


416 


448 


420 


449 


429 


447 


413 


44] 


SD 


27.80 


29.51 


30.81 


22.31 


27.57 


22.42 


28.12 


29.96 


Min. 


337 


388 


335 


407 


343 


396 


333 


379 


Max. 


492 


503 


543 


520 


496 


496 


500 


500 


Skew. 


.02 


-.22 


40 


1.08 


-.13 


-.22 


O11 


-.45 


Kurt. 


a 


46 


84 


3.28 


071 


32 


014 


-.10 
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Similar to previous years, the mean OST score was higher for STEM students with means of 
448 for 5"-grade science and 449 for 7"-grade science compared to 416 and 420, 
respectively. In addition, the range between the minimum and maximum OST scores was 
significantly larger for general education students for all three tests. For example, for fifth- 
grade science the range was 155 points (min.= 337, max. 492) compared to STEM students 
with a range of 115 (min. 388, max. 503). Both the skewness and kurtosis were within 
acceptable limits for all tests for the 2013-2014 school year that indicated a symmetrical data 


distribution and all data were fairly mesokurtic. 


Table 19 displays all descriptive information of OST scores for the 2014-2015 school year. 


Table 19. Descriptive Statistics for 5""- Science, 7'"-Grade Math, 8""-Grade Math, and 8"- 
Grade Science for the 2014-2015 School Year 


Test Mean SD Min. Max. Skew. Kurt. 
Grade 5 Gen. Ed. 704 53.20 376 845 -2.74 14.66 
Science (n= 349) 
STEM ee 40.40 652 820 -.34 -.16 
(n=35 ) 
Grade 7 Gen. Ed. 725 51.29 327 794 -4.95 30.48 
Math (n= 384) 
STEM 759 15.71 730 785 -.10 -.86 
(n= 24) 
Grade 8 Gen. Ed. 732 44.59 373 810 -4.47 31.35 
Math (n= 333) 
STEM 77\ 20.70 733 798 -.52 -.61 
(n=14) 
Grade 8 Gen. Ed. 713 50..80 379 823 -2.02 11.66 
Science (n= 399) 
STEM 759 28.49 720 843 .98 1.54 
(n= 25) 


The mean OST score was higher for STEM students than general education students for all 


tests in that year. In addition, the range between the minimum and maximum OST scores was 
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significantly larger for general education students for all tests. For example, for seventh-grade 
math, the range was extremely large at 467 points (min.= 327, max. 794) compared to STEM 
students with a range of 55 points (min. 730, max. 785). The standard deviations in OST 
scores were all lower for STEM students further demonstrating the lower ranges among 
STEM students. The skewness in OST scores for general education students was outside the 
generally acceptable values ranging from -4.47 to -2.02. Consistently negative skewness to 
the left indicates the mean was less than the median with fewer very low scores. In addition, 
the kurtosis for general education students' scores spanned from 11.66 to 31.55, outside of the 
normal acceptable values indicating the data distribution is leptokurtic with a high ratio of 
scores gravitating toward the mean in a normal distribution. Only 14 STEM participants took 
the 8""-grade OST math test as the remaining students took the algebra end-of-course exam. 


Table 20 displays all descriptive information of OST scores for the 2015-2016 school year. 


Table 20. Descriptive Statistics for 5""_Grade Science, 7"-Grade Math, 8"-Grade Math, and 
8'"-Grade Science for the 2015-2016 School Year 


Test Mean SD Min. Max. Skew. Kurt. 


Grade 5 Gen. Ed. 717 47.10 401 845 -1.55 9.38 
Science (n=332 ) 
STEM 732 48.92 611 827 -.80 .66 
(n=35) 
Grade 7 Gen. Ed. 700 47.75 408 788 -3.17 16.77 
Math (n= 358) 
STEM 738 19.57 708 769 -.02 -1.10 
(n= 24) 
Grade 8 Gen. Ed. 695 50.38 371 774 -4.38 22.61 
Math (n= 320) 
STEM 732 18.78 692 774 225 1.38 
(n=16) 
Grade 8 Gen. Ed. 710 61.42 387 849 -2.21 9.63 
Science (n=378) 
STEM 761 33.40 678 826 -.21 54 
(n= 23) 
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The mean OST score was higher for STEM students than general education students for all 
tests in that year. In addition, the range between the minimum and maximum OST scores was 
significantly larger for general education students for all tests. For example, the range for 
general education students scores in the eighth-grade science test was significantly larger at 
462 points (min.= 387, max. 849) compared to STEM students with a range of 148 points 
(min. 678, max. 826). The standard deviations in OST scores were all lower for STEM 
students except for the 5'"-grade science test with standard deviations for general education 
and STEM students being at 47.10 and 48.92, respectively. The skewness in both OST math 
scores for general education students was outside the generally acceptable values ranging 
from -3.17 to -4.38. Consistently negative skewness to the left indicates the mean was less 
than the median with fewer very low scores. Also, the kurtosis for both OST math scores for 
general education students spanned from 16.77 to 22.61, outside of the normal acceptable 
values indicating the data distribution is leptokurtic with a high ratio of scores gravitating 
toward the mean in a normal distribution. Only 16 STEM participants took the 8'"-grade OST 
math test due to the remaining seven students taking the algebra end-of-course exam. Table 


21 displays all descriptive information of OST scores for the 2016-2017 school year. 


Table 21. Descriptive Statistics for 7"_Grade Math, 8""_Grade Math, and 8""_Grade Science 
for the 2016-2017 School Year 


Test Mean SD Min. Max. Skew. Kurt. 
Grade 7 Gen. Ed. 701 52.43 382 806 3,12 16.00 
Math (n= 342) 
STEM 729 44.99 636 806 -.56 -.32 
(n=37 ) 
Grade 8 Gen. Ed. 701 41.36 420 760 -4.26 27.87 
Math (n=276 ) 
STEM 728 15.42 709 756 .69 -.99 
(n=15) 
Grade 8 Gen. Ed. 71I 49.62 425 811 -2.27 11.13 
Science (n= 362) 
STEM yea 29.8 697 811 -.09 -.66 
(n= 24) 
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The mean OST score was higher for STEM students than general education students for all 
tests in that year. In addition, the range between the minimum and maximum OST scores was 
significantly larger for general education students for all tests. For example, the range for 
general education students scores in the eighth-grade math test was significantly larger at 340 
points (min.= 420, max. 760) compared to STEM students with a range of 47 points (min. 
709, max. 756). The standard deviations in OST scores were all lower for STEM students. 
The skewness in both OST math scores for general education students was outside the 
generally acceptable values ranging from -4.26 to -2.27. Consistently negative skewness to 
the left indicates the mean was less than the median with fewer very low scores. In addition, 
the kurtosis for both OST math scores for general education students spanned from 11.23 to 
27.87, outside of the normal acceptable values indicating the data distribution is leptokurtic 
with a high ratio of scores gravitating toward the mean in a normal distribution. Only 15 
STEM participants took the 8"-grade OST math test due to the remaining nine students 


taking the algebra end-of-course exam. 


Table 22 displays all descriptive information of OST scores for the 2017-2018 school year. 


Table 22. Descriptive Statistics for 7""_Grade Math, 8""_Grade Math, and 8""_Grade Science 
for the 2017-2018 School Year 


Test Mean SD Min. Max. Skew. Kurt. 
Grade 7 Gen. Ed. 709 38.89 400 806 -1.71 12.18 
Math (n= 313) 
STEM 727 42.33 636 801 -.48 -.11 
(n= 36) 
Grade 8 Gen. Ed. 693 49.16 376 766 -4.51 25.16 
Math (n= 252) 
STEM 706 32.40 644 TAT -.70 -.55 
(n= 13) 
Grade 8 Gen. Ed. 715 55.82 385 863 -2.94 15.61 
Science (n= 328) 
STEM 760 49.88 636 868 -.75 91 
(n= 24) 
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The mean OST score was higher for STEM students than general education students for all 
tests in that year. In addition, the range between the minimum and maximum OST scores was 
significantly larger for general education students for all tests. For example, the range for 
general education students scores in the 7th-grade math test was significantly larger at 406 
points (min.= 400, max. 806) compared to STEM students with a range of 165 points (min. 
636, max. 801). The standard deviations in OST scores were all lower for STEM students 
except for the 7'"-grade math test with standard deviations for general education and STEM 
students being at 38.89 and 42.33, respectively. The skewness for both OST 8""-grade tests 
for general education students was outside the generally acceptable values ranging from -4.51 
to -2.94. Consistently negative skewness to the left indicates the mean was less than the 
median with fewer very low scores. In addition, the kurtosis for both OST math scores for 
general education students spanned from 15.61 to 25.16, outside of the normal acceptable 
values indicating the data distribution is leptokurtic with a high ratio of scores gravitating 
toward the mean in a normal distribution. Only 13 STEM participants took the 8'"-grade OST 
math test due to the remaining 11 students taking the algebra end-of-course exam. Table 23 


displays all descriptive information of OST scores for the 2018-2019 school year. 


Table 23. Descriptive Statistics for 7"_Grade Math, 8""_Grade Math, and 8""_Grade Science 
for the 2018-2019 School Year 


Test Mean SD Min. Max. Skew. Kurt. 
Grade 7 Gen. Ed. 709 50.91 376 806 -2.35 12.55 
Math (n=326 ) 
STEM 727 34.11 655 806 .04 .06 
(n= 29) 
Grade 8 Gen. Ed. 703 40.27 373 770 -4.55 32.24 
Math (n=276 ) 
STEM 711 23.18 664 746 -.64 -.14 
(n= 17) 
Grade 8 Gen. Ed. 724 50.87 405 836 -2.09 10.43 
Science (n= 327) 
STEM 761 47.62 651 857 -.60 21 
(n= 34) 
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The mean OST score was higher for STEM students than general education students for all 
tests in that year. In addition, the range between the minimum and maximum OST scores was 
significantly larger for general education students for all tests similarly to all previous years 
in the study. The standard deviations in OST scores were all lower for STEM students. The 
skewness in scores for all tests involving general education students were outside the 
generally acceptable values ranging from -4.55 to -2.09. Consistently negative skewness to 
the left indicates the mean was less than the median with fewer very low scores. In addition, 
the kurtosis for all OST tests taken by general education students spanned from 10.43 to 
32.24, outside of the normal acceptable values indicating the data distribution is leptokurtic 
with a high ratio of scores gravitating toward the mean in a normal distribution. Only 17 
STEM participants took the 8""-grade OST math test due to the remaining students taking the 
algebra end-of-course exam. The number of STEM participants taking the 8'"-grade science 
assessment was 34 students. Although, there were between 25-28 students per grade level 
participating in STEM, students who dropped out of the STEM program at the end of seventh 
grade are identified as STEM students in the study. 


Research Question 2: 

Do students participating in the integrated middle school STEM program demonstrate 
differences in achievement due to the interaction effect of gender, race, socioeconomic 
status, or attendance between the STEM program students and general education 


students? 


Table 24 displays student gender disaggregated by STEM participation for the duration of the 
study. There were 3,035 general education students and 205 STEM students participating in 
the study. The ratio of male to female general education students was almost 1:1 with 49.9 to 
50.1 percent male to female students, respectively. Among the STEM students the ratio of 


male to female was 3:2. 


Among the 3,032 general education participants who provided race information, the majority 
of students were White (77.3%), with the second race indicated as Black (13.1%). Other races 
were represented at significantly lower percentages. Among the 205 STEM participants who 
provided information on race, the majority were also White (87.3%) with other races, such as 
Multiracial (6.3%) and Black (3.9%), representing a much lower percentage. The STEM 


program was less racially diverse than the general education population. 
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Table 24. Student Gender Disaggregated by STEM Participation for the Entire Study 2010- 
2011 through 2018-2019 


Gender Frequency Percent 
Gen. Ed. (n= 3035) Male pep ey 49.9 
Female 1520 50.1 
Total 3035 100 
STEM (n= 205) Male 123 60 
Female 82 40 
Total 205 100 


Table 25 shows the breakdown of student race for students disaggregated by STEM 


participation. 


Table 25. Student Race Disaggregated by STEM Participation for the Entire Study 2010- 
2011 through 2018-2019 


Race Frequency Percent 
Gen. Ed. (n= 3032) American Indian 7 2 
Asian 28 2 
Black 398 13.1 
Hispanic 119 3.9 
Multiracial 133 4.4 
Puerto Rican 2 wl 
White 2345 The 
Total 3032 100 
STEM (n= 205) American Indian 0 0 
Asian 1 a 
Black 8 3.9 
Hispanic 4 2.0 
Multiracial 13 6.3 
Puerto Rican 0 0 
White 179 87.3 
Total 205 100 
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Table 26 displays SES of general education students and STEM students indicating both the 


frequency and percent of students economically and not economically disadvantaged. 


Table 26. Socioeconomic Status (SES) Disaggregated by STEM Participation for 2010-2011 


SES Frequency Percent 
Gen. Ed. (n= 396) Not economically 222 56.1 

disadvantaged 

Economically 174 43.9 

disadvantaged 

Total 396 100 
STEM (n= 24 ) Not economically 16 66.7 

disadvantaged 

Economically 8 33.3 

disadvantaged 

Total 24 100 


The percent of students deemed economically disadvantaged was higher in the general 
education population compared to students in the STEM program for every school year of the 
study. The variable code in SPSS for students deemed not economically disadvantaged was 
SES=0. This created validity errors as missing information was entered as zero during data 
processing. Therefore, the percent of students identified as not economically disadvantaged is 
higher than the state-reported economically disadvantaged percentages that range from 51% 
to 54% depending upon the year. The remaining tables displaying descriptive statistics for 
subsequent years are located in Appendix C due to redundancy and lack of relevance. This is 


discussed as a limitation in Chapter 5. 


Table 27 displays the descriptive statistics for attendance rate of general education students 


and STEM students for the 2010-2011 school year. 
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Table 27. Descriptive Statistics for Student Attendance in the 2010-2011 School Year 


Mean SD Min. Max. Skew. Kurt. 
Gen. Ed. (n= 396) .9570 0341 .79 1.00 -1.32 2.13 
STEM (n= 24) .9470 .0373 85 99 -1.15 62 


There were relatively small differences in the mean attendance rate between general 
education and STEM students with higher attendance among general education students in 
2010-2011 school year. There was a higher mean attendance among STEM students all the 
other remaining years. The remaining tables displaying descriptive statistics for subsequent 
years are located in Appendix B due to redundancy and lack of relevance. This is expanded 


upon in Chapter 5. 


Inter-correlations between Variables 


Preliminary bivariate relationships between STEM participation and student-level variables 
were determined using SPSS (V. 26) and are displayed in Tables 28, 29, and 30. Table 28 
shows the Pearson correlation coefficients between student-integrated STEM participation 
and the student-level coded variables of GENDER and RACE. Out of the 3,237 and 3,240 
students with data for GENDER and RACE, the Pearson bivariate correlation was extremely 
low (r= 0.069 and -.049, respectively) indicating a lack of practical significance. The r 
values, although statistically significant, are inflated due to the very large sample size of 
participants in the study. Therefore, there is no practical significance between these student- 


level variables and STEM participation. 


Table 28. Pearson Correlation between STEM Participation and Student-level Variables 


Independent of School Year 


Variable STEM Sig. (2-tailed) N 

STEM 1 3240 
RACE .069* 3237 
GENDER -.049** 0.005 3240 
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Table 29 shows the Pearson correlation coefficients between student-integrated STEM 
participation and the student-level coded variable of SES by school year. For every school 
year, the number of students with socioeconomic information ranged from 405 for the 2011- 
20112 year to 1253 for the 2012-2013 school year due to the number of participants with 
OST data. The bivariate correlation was extremely low and lacking significance every year 
ranging from -.08 to 0.02. This indicates a lack of correlation between the student-level 


variable of socioeconomic status and STEM participation. 


Table 29. Pearson Correlation between STEM Participation and SES by School Year 


Variable STEM Sig. (2-tailed) N 

Y2010_SES -0.05 0.31 420 
Y2011_SES -0.06 0.227 405 
Y2012 SES -0.071* 0.04 842 
Y¥2013_SES -.059* 0.037 1253 
Y2014 SES -.068* 0.018 1222 
Y2015_ SES -0.048 0.101 1153 
Y2016 SES -0.013 0.728 767 
Y2017_SES 0.02 0.612 621 
Y2018 SES -.080* 0.032 23 


The Pearson correlation coefficient between student integrated STEM participation and the 
student-level coded variable of attendance rate by school year is shown in Table 30. For 
every school year, the number of students with attendance data ranged from 405 for the 2011- 
2012 year to 1,253 for the 2012-2013 school year due to the number of participants with OST 
data. The bivariate correlation was extremely low and of no practical significance every year 
ranging from -.065 to 0.07. This indicates a weak correlation between the student-level 


variable of attendance rate and STEM participation. 
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Table 30. Pearson Correlation between STEM Participation and Attendance by School Year 


Variable STEM Sig. (2-Tailed) N 
Y2010_ ATTEND -0.065 0.183 420 
Y2011_ ATTEND 0.063 0.208 405 
Y2012_ ATTEND 0.021 0535 842 
Y2013_ ATTEND 0.016 0.568 1253 
Y2014 ATTEND 0.05 0.092 1146 
Y2015 ATTEND 0.102** 0.001 1081 
Y2016 ATTEND -0.043 0.271 669 
Y2017_ATTEND 0.017 0.682 601 
Y2018 ATTEND 0.07 0.079 623 


Hierarchical Linear Modeling Related to Research Questions 


The research question addressed in this study is to determine the impact of middle school 
integrated STEM programming on student performance and achievement. Currently, little 
research has been conducted on the impact of STEM programming and instruction on student 
academic performance and achievement. The primary research question to be addressed in 
this research is as follows: 
Does middle school integrated STEM programming positively affect student 
achievement? 
The additional five research questions related to the primary question are stated below with 


the HLM data presented pertaining to each research question. 
Research Question I 


Do students participating in an integrated middle school STEM program demonstrate 


differences in academic achievement in math and science compared to students participating 
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in a traditional general education setting? 


Research question one can be broken down into two sub-questions. 
Sub-question 1: 
Do students participating in an integrated middle school STEM program demonstrate 


differences in academic achievement in both math and science? 


Model 1- Academic Achievement by Year 


The use of HLM to determine the effect of integrated STEM programming on student 
achievement was modeled using different variables at level-1 and level-2. The first model 
used OST score (SCALEDSC) and STEM participation (STEMMARK) at level-1 and the 
assessment type indicated by year (ASSMTTYP) as level-2. Table 31 displays descriptive 
statistics for level-1. There were 8,874 data points at level-1 with a mean scaled score of 
598.94. The test scores are mutually exclusive for individual subjects, grades, and years but 


HLM accounts for this by nesting the data within level-2. 


Table 31. Model 1: Level-1 Descriptive Statistics 


Variable Name N Mean SD Min. Max. 
STEMMARK 8874 0.06 0.24 0.00 1.00 
SCALEDSC 8874 598.94 149.06 314.00 868.00 


Table 32 displays Model 1 descriptive statistics for level-2. There were nine assessment types 


indicated by years of the study. 


Table 32. Model 1: Level-2 Descriptive Statistics 


Variable Name N Mean SD Min. Max. 


ASSMTTYP 9 1.22 0.44 1.00 2.00 


Equation 6 displays the HLM equation at level-1, OST scores are shown as the outcome 


variable (SCALEDSC;j) and STEM participation (STEMMARK;,,) is the level-1 predictor 
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variable. Equation 7 shows level-2 with assessment type (ASSMTTYP)) as the level-2 


predictor variable. 


Level-1 Model 


SCALEDSC;; = Boj + Bij*(STEMMARK;;) + Tj (6) 
Level-2 Model 
Boj = Yoo + Yor*(ASSMTTYP)) + uo; ) 
Bij = Y10 


The mixed model below (Equation 8) substitutes the intercept of the jth level-2 (Bo) from 
Equation 7 into Equation 6 to get the mixed model shown below (Equation 8). The combined 
model contains both the level-1 and level-2 predictors and a term across levels containing 
both random and fixed effects unique to HLM analysis. The analysis of variance (ANOVA) 
model was used to determine the mean achievement scores among both general education 


students and students participating ina STEM program and compare the differences. 


This was performed to measure the variation between student-level and grade-level 
assessment groups. This mixed model, combining both fixed and random effects, was used to 
analyze the relationship between student achievement as a function of STEM programming 
versus general education programming. The proposed mixed model was found to be 
significant in predicting student achievement as a function of the defined level-1 and level-2 


variables as shown below in Table 33 and Table 34. 


Mixed Model 


SCALEDSC; = Yoo + yor ASSMTTYP; - yio*STEMMARK;,, + Uojt Tij (8) 


The final estimation of fixed effects with robust standard errors for Model 1 are shown in 


Table 33. 
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Table 33. Model 1: Final Estimation of Fixed Effects (with Robust Standard Errors) 


Fixed Effect Coefficient | Standard t-ratio Approx. p-value 
error df. 


For INTRCPTI, Bo 
INTRCPT2, yoo 836.77 09.32 8.43 7 <0.001 
ASSMTTYP, yo1 -211.11 49.72 -4.25 7 0.004 
For STEMMARK slope, B; 


INTRCPT2, yio 31.33 229 13.69 8864 <0.001 


Fixed effects were used because the level-2 group was a unique entity and 7 was small 
indicating the number of years (j<10). Robust standard errors were used for both Model | and 
Model 2 due to confidence in the distribution of the dependent variable of assessment type at 
level-2. The overall mean intercept adjusted for student achievement by year (ASSMTTYP) 
for STEM students (STEMMARK) was determined to be 31.3 (INTRCPT2, yi0 ), indicating a 
significant correlation between STEM participation and student achievement. Student 
achievement as indicated by OST scores for a given year, grade, and subject indicate that a 
STEM student was predicted to score 31.3 points higher than general education students. All 
p-values are statistically significant (p S 0.004) supporting the correlation between STEM 
program participation and student achievement. The final estimation of variance shown in 
Table 34 displays the random error associated with the use of the final estimation of fixed 


effects. The random effect at level-1 has a standard deviation of 44.68. 


Table 34. Model 1: Final Estimation of Variance Components 


Random Standard Variance af. \- p-value 
Effect Deviation Component 

INTRCPT1, wo 131.48 17285.64 7 74797.10 <0.001 
level-1, r 44.68 1996.36 


Sub-question 2: 
Do students participating in an integrated middle school STEM program demonstrate 


differences in academic achievement in science? 
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Model 2- Comparing 5"- and 8'-Grade Science 


The goal of the second model was to effectively predict the OST score for students taking 
both the fifth grade and eighth grade science OST tests as a function of STEM participation. 
The model used OST score (SCALEDSC) and STEM participation (STEMMARK) at level-1 
(similar to Model 1) and the assessment type indicated by year (GRADE) as level-2. Table 35 


displays descriptive statistics for both level-1. 


Table 35. Model 1: Level-1 Descriptive Statistics 


Variable Name N Mean SD Min. Max. 
STEMMARK 4048 0.06 0.24 0.00 1.00 
SCALEDSC 4048 562.50 156.94 314.00 868.00 


There were 4,048 data points at level-1 with a mean scaled score of 562.50. The test scores 
are mutually exclusive for individual grades and years but HLM accounts for this by nesting 


the data within level-2. 


The descriptive statistics at level-2 are shown in Table 36. There were nine science 
assessments indicated by the variable GRADE occurring over the course of the study 
beginning in 2010-2011 and ending in 2018-2019. The minimum value was 5 indicating fifth 


grade and the maximum value was 8 indicating eighth grade. 


Table 36. Model 2: Level-2 Descriptive Statistics 


Variable Name N Mean SD Min. Max. 


GRADE 9 6.00 1.50 5.00 8.00 


Equation 9 displays the HLM equation at level-1, OST scores are shown as the outcome 
variable (SCALEDSC;;) and STEM participation (STEMMARK;,,) is the level-1 predictor 
variable. Equation 10 shows level-2 with science assessment (GRADE;) as the level-2 


predictor variable. 
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Level-1 Model 


SCALEDSCjj = Bo; + Bi;*(STEMMARK;) + rij (9) 
Level-2 Model 
Boj = Yoo + Yor*(GRADE)) + uo; (10) 
Bij = Y10 


The mixed model in Equation 11 substitutes the intercept of the jth level-2 (Boj) from 
Equation 10 into Equation 9 to get the mixed model shown below (Equation 11). This model 
was found to be significant in predicting student achievement as a function of the defined 


level-1 and level-2 variables as shown below in Table 37 and Table 38. 


Mixed Model 
SCALEDSC; = Yoo + Yor* GRADE; oe yio*STEMMARK + Ugjt Fij el 1) 


The final estimation of fixed effects with robust standard errors for Model 2 are shown in 


Table 37. 


Table 37. Model 2: Final Estimation of Fixed Effects (with Robust Standard Errors) 


Fixed Effect Coefficient Standard t-ratio Approx. p-value 
error df. 


For INTRCPT1, Bo 


INTRCPT2, yoo 170.67 151.51 1.13 7 0.297 


GRADE, yo, 68.17 18.98 3.59 i; 0.009 


For STEMMARK slope, /; 


INTRCPT2, y70 38.19 3.14 12.17 4038 <0.001 
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The overall mean intercept adjusted for student achievement by year (ASSMTTYP) for 
STEM students (STEMMARK) was determined to be 38.2 (INTRCPT2, yio ), indicating a 
significant correlation between STEM participation and student achievement as evidenced by 
fifth grade to eighth grade science OST scores. Student achievement measured by OST scores 
for fifth and eighth grade science predicted STEM students will score 38.2 points higher than 
general education students. This correlation was stronger than all assessment types used in 
Model 1. All p-values are statistically significant (p S 0.009) supporting the correlation 
between STEM program participation and student achievement in science except for 


INTRECPT2, yoo with a p-value of 0.297. 
The final estimation of variance shown in Table 38 displays the random error associated with 
the use of the final estimation of fixed effects. The standard deviations in the random effect 


associated with level-1, r was 45.42 and an INTRCPT1, wo of 128.88. 


Table 38. Model 2: Final Estimation of Variance Components 


Random Effect Standard Variance af. ‘a p-value 
Deviation Component 

INTRCPT1, wo 128.88 16608.96 fj 25720.24 <0.001 

level-1, 7 45.42 2062.89 


The remaining research question sought to determine if students participating in an integrated 
middle school STEM program demonstrated differences in academic achievement due to the 


interaction effect of student gender, race, socioeconomic status, and attendance. 

Research Question 2 

Do students participating in the integrated middle school STEM program demonstrate 
differences in achievement due to the interaction effect of gender, race, socioeconomic status, 


or attendance between the STEM program students and general education students? 


Previously in this chapter, preliminary bivariate relationships between STEM participation 


and student-level variables of gender, race, socioeconomic status, and attendance were 
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determined and displayed in Tables 28, 29, and 30. Pearson correlation coefficients between 
student achievement and all student-level variables were extremely low indicating no 
significant relationship and correlation between these student-level variables and STEM 


participation. 


Chapter Summary 


Chapter 4 displays the variable and sample descriptions with descriptive statistics related 

to each research question. The descriptive statistical information for OST scores by year are 
displayed indicating the mean, standard deviation, minimum score, maximum score, 
skewness, and kurtosis for each subject, grade level, and year. The inter-correlation between 
variables were shown with preliminary bivariate relationships indicated by Pearson 
correlation coefficients. Two HLM models were created addressing the research questions to 
determine differences in achievement among the multilevel data of students, located in a 
classroom, nested in a grade level, and found within a school.Chapter 5 discusses the 
summary of the findings as they pertain to each research question along with limitations, 


implications for practice, and directions for future practice. 
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CHAPTER 5: IMPLICATIONS FOR STUDENT AND SCHOOL 


EFFECTS ON ACADEMIC ACHIEVEMENT IN A STEM 
PROGRAM 


“Research shows that there is only half as much variation in student achievement 
between schools as there is among classrooms in the same school. If you want your 
child to get the best education possible, it is actually more important to get him [or 
her] assigned to a great teacher than to a great school.” 


Bill Gates (date unknown) 
Summary of Findings 


A summary of findings is explained as it pertains to each research question addressed in this 
study to determine the impact of middle school integrated STEM programming on student 
performance and achievement. The impetus of this research was due to the lack of research 
conducted on the impact of STEM programming and instruction on student academic 
performance, particularly achievement. The summary of findings indicate significant 
differences in student achievement in math and science with students participating in STEM 
performing markedly higher than students in the general education setting. Preliminary 
bivariate relationships between STEM participation and the student-level variables of gender, 
race, socioeconomic status, and attendance were determined to be extremely low indicating 
no significant relationship or correlation between these variables and student achievement. 

An explanation and summary of findings for the proposed research questions are explained 


below. 
The primary research question to be addressed in this research is as follows: 


Does middle school integrated STEM programming positively affect student 


achievement? 
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Research Question 1 
Do students participating in an integrated middle school STEM program demonstrate 
differences in academic achievement in math and science compared to students 


participating in a traditional general education setting? 


The remaining research questions discussed in previous chapters sought to determine if 
students participating in an integrated middle school STEM program demonstrated 
differences in academic achievement due to the interaction effect of student gender, race, 
socioeconomic status, and attendance. These were combined into one research question due 
to the lack of interaction effect found. Therefore, the remaining four research questions in this 


study were combined into one all-encompassing research question shown below. 


Research Question 2 

Do students participating in the integrated middle school STEM program demonstrate 
differences in achievement due to the interaction effect of gender, race, socioeconomic 
status, or attendance between the STEM program students and general education 


students? 
Research Question 1 Findings 


Do students participating in an integrated middle school STEM program demonstrate 
differences in academic achievement in math and science compared to students participating 


in a traditional general education setting? 
Mathematics and Science Achievement 


Past research has determined the impact of STEM education policies and initiatives on 
student achievement having varying degrees of success (Dugger, 2010; Gonzalez & Kuenzi, 
2012; Snyder, 2018; White, 2014). Although Gonzalez and Kuenzi (2012) reported there is 
no single statistic that can fully quantify the success of STEM education on a national, state, 
or local level, this study attempted to gain insight into the impact of STEM education on 
student achievement at a school and student level. Quantifying student achievement via the 
OST, a norm-referenced statewide assessment, is the gold standard for measuring student 


achievement and is the primary indicator of achievement in all previous literature discussed. 
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There were three research studies similar to this study found in recent literature. First, Wade- 
Shepard (2016) investigated the effect of middle school STEM curriculum on both science 
and math achievement scores. The research was conducted among four schools of seventh 
and eighth grade students in Tennessee using the Tennessee Comprehensive Assessment 
Program (TCAP). Instead of performing HLM for data analysis, Wade-Shepherd (2016) used 
two ANCOVAs and the Pearson correlation to determine the strength of the relationship 
between mathematics and science scores of students participating in STEM education classes 
and those that were not participating. The study did not include data analysis of other 
moderators of achievement, such as attendance, demographic information, and teacher 
efficacy. The study found a significant, strong, and positive correlation between test scores of 
students participating in STEM classes compared to those that were not taking STEM classes 


across the four schools (Wade-Shepard, 2016). 


The second research study by Hansen and Gonzalez (2014) investigated the relationships 
between STEM learning principles, such as PBL and student achievement in math and 
science. This mixed methods study included middle school students in North Carolina and 
used a combination of quantitative state assessment and qualitative student survey data. The 
study found specific STEM practices were associated with performance gains in math and 
science. For example, projects and science experiments were associated with higher scores in 
science and the use of technology and computers were associated with higher scores in math. 
In addition, these significant and positive correlations were also found among racial 


minorities (Hansen & Gonzalez, 2014). 


The third research study analyzed both STEM curriculum and PBL strategies on student 
mathematics performance disaggregated by low, middle, and high achieving students to 
determine the degree of effect as a function of student achievement level (Han et al., 2015). 
The study took place in Texas among three high schools with students in the treatment group 
participating in STEM PBL activities once every six weeks over the course of three years. 
This study was similar to the current study as it used HLM to determine the effect of STEM 
PBL activities on students’ mathematics scores accounting for student moderators such as 
student socioeconomic status and race. Han et al. concluded lower achieving students showed 
a statistically significant higher rate of growth on math scores compared to middle and high 
performing students over the course of three years. They also found student race and 


socioeconomic status were strong predictors of student academic achievement (Han et al.). 
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Although student race and socioeconomic status are often correlated with student growth and 
achievement similar to what was reported by Han et al., the present study did not conclude 


similar results to support the claims in previous literature. 


This study was a combination of the above three studies summarized above studying both 
science and math achievement, similar to Wade-Shepard (2016) and Hansen and Gonzalez 
(2014). In addition, this study used HLM and also analyzed the effect of demographic 
moderators, similar to Han et al. (2015). The results of this study indicate through 
comparison of descriptive statistics and HLM analysis that middle school students 
participating in integrated STEM programming scored significantly higher on the OST 


compared to their general education peers. 


In this study, Model 1 used OST score (SCALEDSC) and STEM participation 
(STEMMARK) at level-1 and the assessment type by year (ASSMTTYP) at level-2. 
Descriptive statistics indicated all participants at level-1 (n=8874) had a mean OST score of 
598. The singular level-2 variable (ASSMTTYP) clustered the level-1 participants into nine 
groups (n=9) for each tested year beginning in the school year 2010-2011 through 2018-2019 
creating a longitudinal sample analysis. A mixed model, containing both fixed and random 
effects, combined both the level-1 and level-2 predictors and was found to be significant in 
predicting student achievement as a function of STEM participation for a given tested year 
cluster. The overall mean intercept adjusted for student achievement by year (ASSMTTYP) 
for STEM students (STEMMARK) was determined to be 31.3 points (INTRCPT2, yio ), 
indicating a significant correlation between STEM participation and student achievement. 
Student achievement as indicated by OST scores for a given year, grade, and subject indicate 
that a STEM student is predicted to score 31.3 points higher than general education students. 
All p-values were significantly small (p S 0.004) supporting the correlation between STEM 


program participation and student achievement. 
Science Achievement 


This study used HLM to determine the effect STEM programming had on student 
achievement. Students took the OST science tests only in the fifth and eighth grade. Because 
the STEM program began in the seventh grade for the duration of the study, STEM students 
had two years of STEM programming by the time they took the eighth grade test. The use of 
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HLM to determine student achievement as a function of STEM programming was powerful 


when comparing scores from the fifth to eight grade. 


One study, published by Gabriel and Quinlin (2016), sought to determine the effect of Project 
Lead the Way (PLTW) curriculum on the 2016 Missouri Assessment Plan Science scores of 
fifth and eighth grade students. This study included mid-range socioeconomic students only 
across two school districts comparing test scores of students exposed to the PLTW 
curriculum and those that were not exposed to the curriculum. The study used a basic ¢-test 
and determined there was a significant, positive effect of STEM programming, particularly 


PLTW on student achievement (Gabriel & Quinlin, 2016). 


In this study, Model 2 used OST score (SCALEDSC) and STEM participation 
(STEMMARK) at level-1 (similar to Model-1) and the science assessment type indicated by 
year (GRADE) at level-2. Descriptive statistics indicated all participants at level-1 (n=4048) 
had a mean OST score of 562 points. The singular level-2 variable (GRADE) clustered the 
level-1 participants into nine groups (n=9) for each tested year of science only beginning in 
the school year 2010-2011 through 2018-2019 creating a longitudinal sample analysis. A 
mixed model, containing both fixed and random effects, combined both the level-1 and level- 
2 predictors and was found to be significant in predicting student achievement as a function 
of STEM participation for a given tested year cluster. The overall mean intercept adjusted for 
student achievement by year (ASSMTTYP) for STEM students (STEMMARK) was 
determined to be 38.2 (INTRCPT2, yo ), indicating a significant correlation between STEM 
participation and student achievement in science. Student achievement as indicated by OST 
scores for a given year, grade, and subject indicate that a STEM student is predicted to score 
38.2 points higher than general education students. All p-values are statistically significant (p 
= 0.009) supporting the correlation between STEM program participation and student 
achievement in science except for INTRECPT2, yog with a p-value of 0.297. 


The predictive results of Model 2 indicate through comparison of descriptive statistics and 
HLM analysis, that middle school students participating in integrated STEM programming 
scored significantly higher on the OST in science compared to their general education peers 
scoring above 38.2 points higher on average. The impact of STEM participation on student 


achievement was stronger when comparing science only in Model 2 INTRCPT2, yio = 38.2) 


to 
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compared to both math and science achievement in Model 1 (INTRCPT2, yio = 31.8), 


respectively. 
Research Question 2 Findings 


The remaining research questions discussed in previous chapters sought to determine if 
students participating in an integrated middle school STEM program demonstrated 
differences in academic achievement due to the interaction effect of student gender, race, 
socioeconomic status, and attendance. These were initially separated into four research 
questions by factors; gender; race; socioeconomic status, and attendance in the previous 
chapters. For simplicity and to prevent redundancy, the research questions were combined 
into one research question due to the lack of interaction effect found. The remaining four 
research questions in this study were combined into one all-encompassing research question 


shown below. 


Research Question 2 

Do students participating in the integrated middle school STEM program demonstrate 
differences in achievement due to the interaction effect of gender, race, socioeconomic 
status, or attendance between the STEM program students and general education 


students? 
Student Achievement and Interaction Effects 


Past research indicates mixed results regarding the impact of STEM education due to 
interaction effects such as gender, socioeconomic status, race, and attendance on student 
achievement (Dugger, 2010; Gonzalez & Kuenzi, 2012; Snyder, 2018; White, 2014). The 
most similar study to the present one conducted by Han et al. (2015) concluded student race 
and socioeconomic status were strong predictors of student achievement in math and science. 
In addition, Hansen and Gonzalez (2014) determined positive gains in achievement among 
underrepresented racial minorities. This present study produced inconclusive results 
quantifying the impact of the potential moderators of gender, socioeconomic status, race, and 


attendance. 


= 8 /] 


A Pathway to Success? A Longitudinal Study Using Hierarchical Linear Modeling of Student and 
School Effects on Academic Achievement in a Middle School STEM Program 


Preliminary bivariate relationships between STEM participation and student-level variables 
of gender, socioeconomic status, race, and attendance were determined using Pearson 
correlation coefficients. All Pearson 7 values were extremely low (ranging from -0.068 to 
0.102) indicating no practical significance in the relationship or correlation between these 


student-level variables and STEM participation. 
Limitations 


There were two types of limitations; one type involved the data collection process and human 
factors and the other type was due to the method of data analysis through multilinear 


modeling and HLM. 


The data collection process posed a few limitations. First, there was a substantial amount of 
missing data for the SES of students. Missing data values were initially coded as 0, therefore 
yielding a higher percentage of students deemed not economically disadvantaged. The 
percent range of economically disadvantaged students for the school for each year of the 
study was 51% to 54% (ODE, 2019). However, the economically disadvantaged percent 
range for general education students for the duration of this study was 38.8% to 47.3%, 
respectively. This was significantly lower than reported indicating an overidentification of 
students deemed not economically disadvantaged. Another limitation in data collection was 
several participants were missing attendance data for any given year. This posed substantial 
inaccuracy in determining the effect of attendance as a moderator of student achievement 
supporting inconclusive results. A further limitation in data collection was the self-reporting 


of student race data by parents. 


Student mortality is another limitation of the study. The last year for fifth grade science 
scores was 2014-2015 because the last year of the study was 2018-2019 with those 
participants as eighth grade students. The number of participants for science tests ranged 
from 327 (8'"-grade science for 2018-2019) to 425 (8'"-grade science for 2013-2014) for 
general education students and 15 (8""-grade math for 2016-2017) to 35 (5""-grade science for 
2014-2015) for STEM students. Although, each STEM class consisted of 25-28 students per 
grade level, the number of STEM students with 5""-grade science scores was larger than that 
due to mortality in the program and students identified as STEM students, although they were 


no longer participating. Fortunately, participant mortality was low with only two to three 


to RB. 
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participants per school year. In addition, the number of participants was substantially lower 
for math tests due to eighth grade students taking the OST end-of-course exam in algebra in 


lieu of the 8""-grade math test. 


Lastly, a handful of students dropped out of the STEM program in the seventh grade and 
were counted as STEM students as eighth graders. Therefore, migration of students out of the 
program, although a small number of participants, posed both internal validity and reliability 


threats. 


There were a few limitations due to the research methodology. First, because this research 
was a case study involving only one school there are limitations regarding generalizability 
due to limited external validity. The present study was longitudinal over the course of seven 
years, therefore, it provided a clearer view of the impact of STEM programming on the 
specific population. A second limitation was a limited number of STEM teachers providing 
the treatment of the study. Therefore, it may be difficult to determine if the differences in 
achievement between STEM and general education students were due to STEM 
programming or teacher-related factors. Teacher efficacy, defined by an educator’s own 
belief that students can learn and overcome challenges, is the most important and powerful 
factor influencing student achievement (Hattie, 2009). Because the STEM program contained 
few teachers, the impact of the treatment on student performance may have been due to 
individual teacher efficacy. The influence of teacher collective efficacy on student 
achievement had a reported effect size of 1.57, more than double the effect size of other 
reported impactful factors, such as programming (Hattie, 2009). In regard to teacher efficacy 
and STEM programming, specifically, Chai et al. (2019) found teachers’ comfortability with 
technological pedagogical knowledge to be the strongest predictor of teacher efficacy among 


STEM educators. 


The last limitation related to the methodology of the study is many STEM students were 
already high achieving prior to participating in the program. This is evidenced by the mean 
OST scores in science for fifth grade STEM students being larger than the mean of general 
education students for all tests in the study. Therefore, the impact of STEM programming 
may be more difficult to determine due to students already being higher achieving on average 


prior to participating in the STEM program in seventh and eighth grades. 
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Implications for Practice 


The findings indicate integrated STEM programming had a positive impact on middle school 
students and increases both science and math performance. This has powerful implications 
for educational leaders, particularly with changes in legislation that gave more power to local 
school districts and increased funding under ESSA (Achieve, 2015; ESSA, 2015). Therefore, 
it is important educational leaders are aware of the impact integrated STEM programming 
and PBL has on student achievement. In addition, due to the increase in funding for STEM 
education there are many STEM curriculum companies promising quality technological 
curriculum, such as PLTW. This research will assist educational leaders in making the 
appropriate fiscal decisions regarding the purchasing of effective STEM programming and 
professional development for teachers. Bybee (2013) discussed major challenges specific to 
STEM education stating these are related to the misunderstandings surrounding the definition 
of STEM education and the qualifications of STEM literacy. The themes related to reform 
according to Bybee (2013) include: 

- focusing on global challenges with citizen understanding, 

- changing societal views of environmental issues, 

- identifying 21st-century skills in the workforce, and 


- challenges facing national security. 


School leaders play an important and foundational role in ensuring the implementation and 
sustainability of STEM reform (Bybee, 2013; Waight et al., 2018). The impact STEM 
education and technological literacy has on challenges facing all citizens around the world, 
such as fighting a global pandemic and world-wide climate change, is important. Decisions 
made by legislators and school leaders are crucial in ensuring the implementation and 


sustainability of STEM reform (Bybee, 2013; Waight et al.2018). 


Research findings are continuing to support a strong impact of integrated STEM 
programming on student achievement at the middle school level. The interdisciplinary 
method of teaching STEM provides students learning concepts and skills from two or more 
disciplines that are tightly linked to deepen knowledge and skills (Kaufman et al., 2003). 
Other methods of teaching STEM previously discussed by Kaufman et al. include: 
disciplinary, multidisciplinary, and transdisciplinary. An implication of this research would 


be to determine the effectiveness of other STEM methods on student achievement. 
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There were two implementation dichotomies discussed previously: isolation versus 
integration and “Some STEM for All” versus “All STEM for Some”. This research further 
demonstrates the success of STEM on student achievement using the integration model, a 
method supported by many in previous research literature (Dugger, 2010; Sanders, 2009). In 
addition, this research analyzed the success of a STEM program model utilizing the “All 
STEM for Some” approach described and supported by Atkinson (2012) and Elrod et al. 
(2012) particularly at higher grade levels. 


The argument among researchers is not whether integrated STEM education is effective at 
the elementary, middle, or high school level, but rather which grade level is most impactful 
for the introduction of STEM practices on student achievement, and subsequently, future 
success. Both the elementary and high school years have been shown to be powerful in regard 
to shaping students’ perceptions of their learning and impact on the development of 
integrated STEM education practices. Somewhere between the two may be the “goldilocks” 
zone, the middle school years, when students are beginning to develop attitudes and beliefs 
regarding their abilities in STEM and possibilities of future careers (Christensen et al., 2015). 
This research study supports the positive impact STEM programming has during the middle 


school years. 


The National Middle School Association (NMSA, 2010) recommended integrated STEM 
curriculum and instruction at the middle school level as it offers engaging and holistic 
instruction for all learners with studies finding the integration of mathematics and science 
having a positive influence on students’ attitudes toward school, their motivation to learn, and 
academic performance. Moreno et al. (2016) remarked, “The middle school years are a 
crucial time for cultivating students' interest in and preparedness for future STEM careers” 
(p. 889). Middle school is a pivotal time for students as their viewpoints on education and 
careers are greatly impacted by their environment and their focus shifts to future careers. This 
research further supports the importance of middle school STEM in shaping students' 


perspectives on future careers. 
Considerations for Future Practice and Research 


There are several considerations regarding directions for future practice and research 


expanding upon this research study. First, an extension of this present study would be to 
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determine the differences in OST scores among low, middle, and high achieving students as a 
function of STEM participation. Han et al. (2015) found low achieving students exposed to 
STEM programming experienced the most growth in achievement compared to their middle 
and higher achieving peers. Disaggregating student achievement by performance level would 
gain deeper information into the impact of STEM on student achievement on different levels 


of learners. 


Another consideration for future research would be to determine teachers’ individual self- 
efficacy for the few educators providing the STEM programming to students in the study. In 
addition, the quality of STEM and PLTW training should be researched. An expansion upon 
this would be to determine the technological pedagogical knowledge of the teachers 
providing STEM programming to determine their individual impact on student achievement. 
Han et al. (2015) conducted an analysis determining the different types of content knowledge 
necessary to be a successful STEM educator. This model called the Technological 
Pedagogical Content Knowledge (TPACK) framework assesses teachers’ self-efficacies to 
determine their impact on student achievement. Collecting TPACK information and survey 
data on STEM teachers would strengthen the results of the impact integrated STEM 
education has on student achievement. The influence of individual teachers is often more 
pronounced than the collective influence of a school or program. Bill Gates, a businessman, 
software developer, and philanthropist summarized the power of teacher impact stating: 
Research shows that there is only half as much variation in student achievement 
between schools as there is among classrooms in the same school. If you want your 
child to get the best education possible, it is actually more important to get him [or her] 


assigned to a great teacher than to a great school. (Gates, n.d.) 


The variability of student performance as a function of STEM participation may have more to 
do with the quality of the educator than the STEM programming itself. Future research needs 
conducted to determine the depth of teacher efficacy on student achievement in STEM 


education. 


This research study was conducted to determine the effectiveness of integrated STEM 
education on student achievement and, subsequently, student future success. The need for 
STEM education was ignited by the Space Race of the 1950s and gained momentum due to A 
Nation at Risk and the development of the AAAS in the 1980s. STEM education is resurging 
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and breathing new life due to urgent needs in medical, scientific, and technological 
advancements prompted by the recent coronavirus pandemic. This research on the importance 
of STEM education will not only create a pathway to success for students but shape 


competent adults able to carry the nation through an unforeseeable future. 
Conclusion 


This longitudinal study reported OST scores along with demographic factors such as gender, 
socioeconomic status, race, and attendance to determine the impact of STEM programming 
on specific populations of middle school students. The use of multilevel, statistical analysis 
through HLM determined integrated STEM programming had a significant, positive effect on 
student achievement in both math and science, and an even stronger impact isolating science 
achievement by itself. The predictive results of HLM analysis determined STEM students 
scored significantly higher on the OST in science and math combined scoring 31.8 points 
higher on average and 38.2 points higher in science compared to their general education 
peers. No interaction effects were determined between STEM participation and gender, 
socioeconomic status, student race, and attendance rate. More research needs conducted on 
the impact of teacher efficacy, individual classroom influences, and quality of STEM training 
for educators. The research results indicate that integrated STEM programming in middle 
school had a positive effect on student achievement indicating it is indeed a pathway to 


success. 
Chapter Summary 


A summary of findings is explained as it pertains to each research question addressed in the 
study to determine the impact of middle school integrated STEM programming on student 
performance and achievement. The results indicate significant differences in student 
achievement in math and science with students participating in STEM performing markedly 
higher than students in the general education setting. Preliminary bivariate relationships 
between STEM participation and the student-level variables of gender, race, socioeconomic 
status, and attendance were determined to be extremely low indicating no practical significant 
relationship or correlation between these variables and student achievement.The predictive 
results of HLM analysis determined STEM students scored significantly higher on the OST 


in science and math combined scoring 31.8 points higher on average and 38.2 points higher 
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in science compared to their general education peers. No interaction effects were determined 
between STEM participation and gender, socioeconomic status, student race, and attendance 
rate. More research needs conducted on the impact of teacher efficacy, individual classroom 
influences, and quality of STEM training for educators. The research results indicate that 
integrated STEM programming in middle school had a positive effect on student achievement 


indicating it is indeed a pathway to success. 


Citation 


Chine, D. R. (2022). Implications for Student and School Effects on Academic Achievement 
in a STEM Program. In J. Johnson & A. Kaban (Eds.), 4 Pathway to Success? A 


Longitudinal Study Using Hierarchical Linear Modeling of Student and School Effects on 
Academic Achievement in a Middle School STEM Program (pp. 103-115). ISTES 


Organization. 


7 A Pathway to Success? A Longitudinal Study Using Hierarchical Linear 
| S te S Modeling of Student and School Effects on Academic Achievement in 


C@ a Middle School STEM Program 


www.istes.org 


REFERENCES 


21st Century STEM for Girls and Underrepresented Minorities Act of 2019. Cong. § 116 
(2019). 

AAAS Editors. (2020). About AAAS. American Association for the Advancement of 
Science. Retrieved November 20, 2020. https://www.aaas.org/mission 

Abdul Rahman, M., Zakaria, M. S., & Din, R. (2021). Design Learning Module Fundamental 
of Computer Science: Evaluation of Content Validity. International Journal of 
Research in Education and Science  (IJRES), 7(4), 1104-1116. 
https://doi.org/10.46328/ijres.2346 

Academic Competitiveness Council. (2007). Report of the academic competitiveness council. 
U.S. Department of Education. 

Achieve. (2017). Leveraging ESSA to promote science and STEM education in states. 
https://www.achieve.org/files/Achieve STEMreport7.12.17.pdf 

Achievement growth. (n.d.) Achievement growth. https://www.edglossary.org/achievement- 
growth/ 

Ansell, S. (2011, July 7). Achievement gap. Education Week. 
https://www.edweek.org/ew/issues/achievement-gap/ 

Ashford, S., Lanehart, R., Kersaint, G., Lee, R., & Kromrey, J. (2016). STEM pathways: 
Examining persistence in rigorous math and science course taking. Journal of Science 
Education and Technology, 25(6), 961-975. https://www.jstor.org/stable/45 151299 

Asigigan, S. I. & Samur, Y. (2021). The Effect of Gamified STEM Practices on Students’ 
Intrinsic Motivation, Critical Thinking Disposition Levels, and Perception of Problem- 
Solving Skills. /nternational Journal of Education in Mathematics, Science, and 
Technology (IJEMST), 9(2), 332-352. https://doi.org/10.46328/1jemst. 1157 

Association of American Universities. (2006). National Defense Education and Innovation 


Initiative. https://files.eric.ed.gov/fulltext/ED505806.pdf 


$$ {tts 


A Pathway to Success? A Longitudinal Study Using Hierarchical Linear Modeling of Student and 
School Effects on Academic Achievement in a Middle School STEM Program 


Atkinson, R. (2012). Why the current education reform strategy won’t work. Issues in 
Science and Technology, 28(3), 29-36. https://www.jstor.org/stable/433 15667 

Baharuddin, Ampera, D., Fibriasari, H., Sembiring, M. A. R., & Hamid, A. (2021). 
Implementation of Cloud Computing System in Learning System Development in 
Engineering Education Study Program. International Journal of Education in 
Mathematics, Science, and Technology (IJEMST), 9(4), 728-740. 
https://doi.org/10.46328/ijemst.2114 

Bandura, A. (1977). Self-efficacy: The exercise of control. W. H. Freeman. 

Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. 
Prentice-Hall. 

Banning, J., & Folkestad, J. (2012). STEM education related dissertation abstracts: A 
bounded qualitative meta-study. Journal of Science Education and Technology, 21(6), 
730-741. https://www.jstor.org/stable/4 1674501 

Barakos, L., Lujan, V., & Strang, C. (2012). Science, technology, engineering, mathematics 
(STEM): Catalyzing change amid the confusion. RMC Research Corporation, Center of 
Instruction. 

Batdi, V., Talan, T., & Semerci, C. (2019). Meta-Analytic and Meta-Thematic Analysis of 
STEM Education. Jnternational Journal of Education in Mathematics, Science and 
Technology (IJEMST), 7(4), 382-399. 

Beede, D., Julian, T., Langdon, D., McKittrick, G., Khan, B., & Doms, M. (2011). Women in 
STEM: A gender gap to innovation. Economics and Statistics Administration, 4(11), 1- 
11. https://ssrn.com/abstract=1964782 

Behizadeh, N. (2014). Enacting problem-posed education through project-based learning. The 
English Journal, 104(2), 99-104. https://jstor.org/stable/24484422 

Benek, I. & Akcay, B. (2019). Development of STEM Attitude Scale for Secondary School 
Students: Validity and Reliability Study. International Journal of Education in 
Mathematics, Science and Technology (IJEMST), 7(1), 32-52. 
DOI:10.18404/ijemst.509258 

Berland, L. (2013). Designing for STEM integration. Journal of Pre-College Engineering 
Education Research, 3(1), 22-31. https://doi:10.7771/2157-9288.1078 

Bhattacharya, D., Guzey, S., Miller, C., & Moore, T. (2015). Artificial floating islands: an 
integrated STEM unit. Science Scope, 38(9), 63-69. 
https://www.jstor.org/stable/43691295 


$M it 


References 


Boyd, D., Grossman, P., Lankford, H., Loeb, S., & Wyckoff, J. (2009). Teacher preparation 
and student achievement. Educational Evaluation and Policy Analysis, 31(4), 416-440. 
https:// doi:10.3102/0162373709353129 

Bradley, V. M. (2021). Learning Management System (LMS) Use with Online Instruction. 
International Journal of Technology in Education (ITE), 4(1), 68-92. 
https://doi.org/10.46328/1jte.36 

Bybee, R. (1997). Achieving scientific literacy. Heinemann. 

Bybee, R. (2013). The case for STEM education: Challenges and opportunities. NSTA Press. 

Canton, N., Gale, B. (Producers), & Zemeckis, R. (Director) (1985). Back to the Future 
[Motion Picture]. 

Chai, C., Jong, M., Yin, H., Chen, M., and Zhou W. (2019). Validating and modelling 
teachers’ technological pedagogical content knowledge for integrative science, 
technology, engineering and mathematics education. Journal of Education Technology 
and Society. 22(3). 61-73. 

Chine, D. R. & Larwin, K. H. (2022). Using HLM to Determine a STEM Programs Impact 
on Middle School Academic Achievement. International Journal of Research in 
Education and Science (IJRES), 8(1), 1-17. https://doi.org/10.46328/ijres.2607 

Christensen, R., Knezek, G., & Tyler-Wood, T. (2015). Alignment of hands-on STEM 
engagement activities with positive STEM dispositions in secondary math students. 
Journal of Science Education and _ Technology, 24(6), 898-909. 
https://www.jstor.org/stable/43867753 

Cohen, A. (2020). 5 Reasons why STEM career training should start in middle school. 
https://everfi.com/insights/blog/5-reasons-stem-career-training-start-middle-school/ 

Connors-Kellgren, C., Parker, C., Blustein, D., & Barnett, M. (2016). Innovations and 
challenges in project-based STEM education: Lessons from ITEST. Journal of Science 
Education, 25(6), 825-832. https://www.jstor.org/stable/45 151289 

Constructivism (philosophy of education). (2020, April 15). In Wikipedia. 
https://en.wikipedia.org/w/index.php?title=Constructivism_ (philosophy of education) 
&oldid=95 1091236 

Crandall, R. (1993). The continuing decline of manufacturing in the rust belt. Brookings 
Institution. 

Crowe, S., Cresswell, K., Robertson, A., Huby, G., Avery, A., & Sheikh, A. (2011). The case 
study approach. BMC Medical = Research —_ Methodology, 11(100). 
https://doi.org/10.1186/1471-2288-11-100 


mir mm 


A Pathway to Success? A Longitudinal Study Using Hierarchical Linear Modeling of Student and 
School Effects on Academic Achievement in a Middle School STEM Program 


Cetin, A. (2021). Investigation of the Relationship between the STEM Awareness and 
Questioning Skills of Pre-Service Teachers. International Journal of Research in 
Education and Science (IJRES), 7(1), 65-81. https://doi.org/10.46328/ 1jres.1171 

DARPA Editors. (2020). About DARPA. https://www.darpa.mil/about-us/about-darpa 

Dedrick, R., Ferron, J., Hess, M., Hogarty, K., Kromrey, J., Lang, T., Niles, J., & Lee, R. 
(2009). 

Dewey, J. (1897). My pedagogic creed. School Journal, 54(1), 77-80. 

Dewey, J. (1904). The relation of theory to practice in education of teachers. National 
Society for the Scientific Study of Education. University of Chicago. 

Doerschuk, P., Bahrim, C., Daniel, J., Kruger, J., Mann, J., & Martin, C. (2016). Closing the 
gaps and filling the STEM pipeline: A multidisciplinary approach. Journal of Science 
Education and Technology, 25(4), 682-695. https://www.jstor.org/stable/43 867815 

Doganca Kucuk, Z., Yabas, D., Boyaci, H. S., & Corlu, M. S. (2021). The Impact of the 
earlySTEM Program on Teacher and Student Outcomes: The Role of Teachers’ 
Involvement in the Program Development. International Journal of Education in 
Mathematics, Science, and Technology (IJEMST), 9(3), 371-405. 
https://doi.org/10.46328/ijemst. 1279 

Donahoo, J. (2007). Collective efficacy: How educators [sic] beliefs impact student learning. 
Corwin. 

Dooley, M., Payne, A., Steffler, M., & Wagner, J. (2017). Understanding the STEM path 
through high school and into university programs. Canadian Public Policy, 43(1), 1-16. 
https://www.jstor.org/stable/10.2307/90001500 

Dugger, W. (2010). Evolution of STEM in the United States. International Technology and 
Engineering Educators Association. 
https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.476.5804&rep=rep 1 &type= 
pd 

Elrod, S., Sawyer, K., & Van Lint, V. (2012). Better STEM for all. Issues in Science and 
Technology, 28(4), 11-15. https://www.jstor.org/stable/433 15632 

Er, Z., Artut, P. D., & Bal, A. P. (2022). Investigation of the Relationship between Estimation 
Skill, Estimation Skill Self-efficacy, and Academic Achievement of Secondary School 
Students. International Journal on Social and Education Sciences (IJonSES), 4(1), 149- 
163. https://doi.org/10.46328/ijonses.292 


$f us }_—____________ 


References 


Etim, J. S., Etim, A. S., & Blizard, Z. D. (2020). Class Size and School Performance: An 
Analysis of Elementary and Middle Schools. International Journal on Studies in 
Education (IJonSE), 2(2), 66-77. 

Every Student Succeeds Act (ESSA). (2017). What does the next 10 years look like in STEM 
education: An overview of the federal education law and federal funding for STEM 
initiatives. static.nsta.org/pdfs/ESSA Overview.pdf 

Funk, C., & Parker, K. (2018a). Women and men in STEM often at odds over workplace 
equity. Pew Research Center analysis of 2014-2016 American Community Survey 
(IPUMS). 

Funk, C., & Parker, K. (2018b). Blacks and Hispanics underrepresented across most STEM 
job clusters. Pew Research Center analysis of 2-14-2016 American Community 
Survey (IPUMS). 

Giasi, T. (2018). Post-program implementation of integrated STEM instruction [Doctoral 
dissertation, The Ohio State University]. 
https://etd.ohiolink.edu/!etd.send_file?accession=osu152568809 1215031 &disposition= 
inline 

Gill, J. (2003). Hierarchical linear models. In Kimberly Kempf-Leonard (Ed.), Encyclopedia 
of social measurement (pp. 1-5). Academic Press. 

Gonzalez, G., Siler-Evans, K., Hunter, G., & Baird, M. (2016). Wages, employment, and 
STEM education in Ohio, Pennsylvania, and West Virginia: Report No. 1. In Wages, 
Employment, and STEM Education in Ohio, Pennsylvania, and West Virginia. Report 
No. 1. RAND Corporation. https://www.jstor.org/stable/10.7249/j.cttl btc11x.1 

Gonzalez, G., Siler-Evans, K., Hunter, G., & Broten, N. (2017). Wages, employment, and 
STEM education in Ohio, Pennsylvania, and West Virginia. Report No. 2. RAND 
Corporation. https://doi.org/10.7249/RR1863 

Gonzalez, H., & Kuenzi, J. (2012). Science, technology, engineering, and mathematics 
(STEM) education: A primer. Congressional Research Service, 7(5700), 1-34. 
https://fas.org/sgp/crs/misc/R42642.pdf 

Haggbloom, S., Warnick, R., Warnick, J., Jones, V., Yarbough, G., Russell, T., Borecky, C., 
McGahhey, R., Powell, J., Beavers, J., & Monte, E., (2002). The 100 most eminent 
psychologists of the 21st century. Review of General Psychology, 6(2), 139-152. 
https:/www.researchgate.net/publication/228079436_The_100 Most _Eminent_Psychol 


ogists of the 20th Century 


tO 


A Pathway to Success? A Longitudinal Study Using Hierarchical Linear Modeling of Student and 
School Effects on Academic Achievement in a Middle School STEM Program 


Han, S., & Buchmann, C. (2016). Aligning science achievement and STEM expectations for 
college success: A comparative study of curriculum standardization. The Russell Sage 
Foundation Journal of _ the Social Sciences, 2(1), 192-211. 
https://www.jstor.org/stable/10.7758/rsf.2016.2.1.09 

Han, S., Capraro, R., Capraro M. (2015). How science, technology, engineering, and 
mathematics (STEM) project-based learning (PBL) affects high, middle, and low 
achievers differently: the impact of student factors on achievement. /nternational 
Journal of Science and Mathematics Education. 13(1). 1089-1113. 

Handelson, J., & Smith, M. (2016). Educate to innovate. 
https://obamawhitehouse.archives. gov/issues/education/k-12/educate-innovate 

Hansen, M., & Gonzalez, T. (2014). Investigating the relationship between STEM learning 
principles and student achievement in math and science. American Journal of 
Education, 120(2), 139-171. https://www.jstor.org/stable/10.1086/674376 

Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to 
achievement. Routledge. 

Hebebci, M.T. (2021). Investigation of Teacher Opinions on Stem Education. In M. Shelley, 
I. Chiang, & O. T. Ozturk (Eds.), Proceedings of ICRES 2021--International 
Conference on Research in Education and Science (pp. 56-72). Monument, CO, USA: 
ISTES Organization. Retrieved from www.2021.icres.net/proceedings/25 

Hebebci, M. T. (2019). The impacts of science, technology, engineering and mathematics 
applications on middle school students' academic achievement, scientific creativity and 
attitudes. Unpublished Phd Thesis, Necmettin Erbakan University, Konya, Turkey. 

Hildebrand, D. (2018). John Dewey. The Stanford Encyclopedia of Philosophy (Winter 2018 
ed.) Stanford University. https://plato.stanford.edu/entries/dewey 

History.com Editors. (2010). The Space Race. https://www.history.com/topics/cold- 
wat/space-race 

Honey, M., Pearson, G., & Schweingruber, H. (Eds.). (2014). STEM integration in K-12 
education: Status, prospects, and an agenda for research. Committee on Integrated 
STEM Education; National Academy of Engineering; National Research Council. 

ITEA (2000/2002/2007). Standards for technological literacy: Content for the study of 
technology. https:// iteea.org/File.aspx?id=42547 

Kartal, T. & Dilek, I. (2021). Developing Pre-service Elementary Science Teachers’ Science 
Teaching Efficacy Beliefs through Microteaching by Promoting Efficacy Sources. 


ADA tt 


References 


International Journal on Social and Education Sciences (IJonSES), 3(4), 710-731. 
https://doi.org/10.46328/ijonses. 124 

Kaufman, D., Moss, D., & Osborn, T. (Eds.) (2003). Beyond the boundaries: A trans- 
disciplinary approach to learning and teaching. Praeger. 

Kennedy, T., & Odell, M. (2014). Engaging students in STEM education. Science Education 
International, 3(25), 246-258. https://www.icaseonline.net/sei/september2014/p1.pdf 

Kier, M., Blanchard, M., & Albert, J. (2014). Connecting students to: STEM careers. Science 
Scope, 37(6), 72-76. https://www.jstor.org/stable/4369 1175 

Kim, Y. R., Park, M. S., & Tjoe, H. (2021). Discovering Concepts of Geometry through 
Robotics Coding Activities. International Journal of Education in Mathematics, 
Science, and Technology (IJEMST), 9(3), 406-425. 
https://doi.org/10.46328/ijemst.1205 

Lawanto, O., Santoso H., & Liu, Y. (2012). Understanding of the relationship between 
interest and expectancy for success in engineering design activity in grades 9-12. 
Journal of — Educational Technology & _ Society, 15(1), 152-161. 
https://wwjstor.org/stable/10.2307/jeductechsoci.15.1.152 

Lesseig, K., Slavit, D., & Holmlund, N. (2017). Jumping on the STEM bandwagon: How 
middle grade students and teachers can benefit from STEM experiences. Middle School 
Journal, 48(3), 15-24. https://www.jstor.org/stable/44320367 

Lindsay, K. G. (2020). A Multilevel Binary Logistic Regression Model of Success in 
Anatomy and Physiology I: A Retrospective Analysis. International Journal of 
Research in Education and Science (IJRES), 6(2), 361-368. 

Locke, E. (2009). Proposed model for a stream-lined, cohesive, and optimized k-12 STEM 
curriculum with focus on engineering. The Journal of Technology Studies, 35(2), 23- 
25. https://www.jstor.org/stable/10.2307/jtechstud.35.2.23 

Lounsbury, J. H. (2010). This we believe: Keys to educating young adolescents. Taylor & 
Francis. 
https://scholar.google.com/scholar?hl=en&as_sdt=0%2C36&as_vis=1&q=This+wetbe 
lieve%3A+Keys+tot+educating+young++adolescents&btnG= 

Lynch, S., Burton, E., Behrend, T., House, A., Ford, M., Spillane, N., Matray, S., Han, E., & 
Means, B. (2017). Understanding inclusive STEM high schools as opportunity 
structures for underrepresented students: Critical components. Journal of Research in 
Science Teaching, 55(1), 712-748. 

Macalalag, A. Z., Sahin, I., Johnson, J., & Bicer, A. (Eds.). (2022). Internalization of STEM 


my 2 Re 


A Pathway to Success? A Longitudinal Study Using Hierarchical Linear Modeling of Student and 
School Effects on Academic Achievement in a Middle School STEM Program 


Education. ISTES Organization. 

Mahoney, M. (2009). Students’ attitudes toward STEM: Development of an instrument for 
high school STEM-based programs [Doctoral dissertation, The Ohio State University]. 
https://etd.ohiolink.edu/!etd.send_file?accession=osu1250264697&disposition=inline 

Mahoney, M. (2010). Students’ attitudes toward STEM: Development of an instrument for 
high school STEM-based programs. The Journal of Technological Studies, 36(1), 24- 
34. https://www.jstor.org/stable/43604248 

Maltese, A., & Tai, R. (2011). Pipeline persistence: Examining the association of educational 
experience with earned degrees in STEM among U.S. students. Science Education 
Policy, 95(1), 877-907. https://doi:10.1002/sce.20441 

Marco-Bujosa, L. (2021). Prospective Secondary Math Teachers Encountering STEM in a 
Methods Course: When Math is More Than “Just Math”. /nternational Journal of 
Technology in Education (IJTE), 4(2), 247-286. https://doi.org/10.46328/ijte.41 

Marra, R., Jonassen, D., Palmer, B., & Luft, S. (2014). Why problem-based learning works: 
Theoretical foundations. Journal of Excellence in College Teaching, 25(3&4), 221-238. 

Masters of Gravity (n.d.). Teacher Guide. Western Reserve Public Media. 
https://westernreservepublicmedia.org/gravity/gravity.pdf 

Matthews, M. (1998). Constructivism in science education: A philosophical examination. 
Springer Science & Business Media. 

McClure, E. (2017). More than a foundation: Young children are capable of STEM learners. 
National Association for the Education of Young Children, 72(5), 83-89. 
https://www.jstor.org/stable/10.2307/90015862 

McClure, E., Guernsey, D., Clements, S., Bales, J., Nichols, N., Kendall-Taylor, & Levine, 
M. (2017). STEM starts early: Grounding science, technology, engineering, and math 
education in early childhood. The Joan Ganz Cooney Center at Sesame Workshop. 
www.joanganzcooneycenter.org/wp- 
content/uploads/2017/01/jgcc_stemstartsearly_final. 

McGinnis, J., Kramer S., Shama, G., Graeber, A., Parker, C., & Watanabe, T. (2002). 
Undergraduates’ attitudes and beliefs about subject matter and pedagogy measured 
periodically in a reform-based mathematics and science teacher preparation program. 
Journal of Research in Science Teaching, 39(8), 713-737. 

McGinnis, P. (2017). STEM integration: A _ tall order. Science Scope, 41(1). 
https://www.jstor.org/stable/26389094 


AA i 


References 


Minken, Z., Macalalag, A., Clarke, A., Marco-Bujosa, L., & Rulli, C. (2021). Development 
of Teachers’ Pedagogical Content Knowledge during Lesson Planning of 
Socioscientific Issues. International Journal of Technology in Education (IJTE), 4(2), 
113-165. https://doi.org/10.46328/ijte.50 

Moreno, N., Tharp, B., Vogt, G., Newell, A., & Burnett, C. (2016). Preparing students for 
middle school through after-school STEM activities. Journal of Science Education and 
Technology, 25(6), 889-898. https://www.jstor.org/stable/45 151294 

Morrison, J. (2006). TIES STEM education monograph series: Attributes of STEM education. 
Teaching Institute for Essential Science. NASA. (2020). Apollo 11 mission overview. 
https:/www.nasa.gov/mission_pages/apollo/missions/apollo1 1 .html 

Multilevel modeling: A review of methodological issues and applications. Review of 
Educational Research, 79(1), 69-102. https://doi.org/10.3 102/0034654308325581 

National Commission on Excellence in Education. (1983). A nation at risk: The imperative to 
education reform. https://www2.ed.gov/pubs/NatAtRisk/index.html 

National Middle School Association. (2010). This we believe: Keys to educating 
young adolescents. 4"" edition. National Middle School Association. 

National Science Board. (2019). Science and engineering indicators 2020. Elementary and 
secondary mathematics and science education. NSB-2019-6. 
https://ncses.nsf.gov/pubs/nsb20196 

No Child Left Behind Act. (2020, April 2). In Wikipedia. 
https://en.wikipedia.org/w/index.php?title=No_Child_Left_Behind_Act&oldid=94873 1 
164 

Nola, R., & Irzik, G. (2006). Philosophy, science, education and culture. Springer Science & 
Business Media. 

Noonan, R. (2017). Women in STEM: 2017 Update (Issue Brief No.06-17). Office of the 
Chief Economist, Economics and Statistics Administration, U.S. Department of 
Commerce. https://www.esa.gov/reports/women-stem-2017-update 

Obama, B. (2013, February 11). STEM for All [Speech]. Whitehouse archives. 
https://obamawhitehouse.archives.gov/blog/2016/02/1 1/stem-all 

Ohio Department of Education. (2014, July). Ohio achievement assessments statistical 
Summary. https://education.ohio.gov/Topics/Testing/Testing-Results/Results-for- 
Ohios-State-Tests 


A Pathway to Success? A Longitudinal Study Using Hierarchical Linear Modeling of Student and 
School Effects on Academic Achievement in a Middle School STEM Program 


Ohio Department of Education. (2015, November). Spring 2015 ELA and math results’ 
Highlights. https://education.ohio.gov/Topics/Testing/Testing-Results/Results-for- 
Ohios-State-Tests 

Ohio Department of Education. (2018). Ohio's learning standards for engineering and 
science technologies and engineering technologies. 
https://education.ohio.gov/getattachment/Topics/Career-Tech/Engineering-and- 
Science-Technologies-Career-Field/2018-Eng-Manuf NO-DOK.pdf.aspx?lang=en-US 

Ohio Department of Education. (2019). [Unknown] school report card. Ohio School Report 
Cards. https://reportcard.education.ohio. gov/district/overview/048298 

Ohio Department of Education. (2019). Ohio's learning standards for mathematics. 
https://education.ohio.gov/Topics/Learning-in-Ohio/Mathematics/Ohio-s-Learning- 
Standards-in-Mathematics 

Ohio Department of Education. (2019). Ohio's learning standards for science. 
https://education.ohio.gov/Topics/Learning-in-Ohio/Science 

Ohio Department of Education. (2019). Ohio's learning standards for technology. 
https://education.ohio.gov/Topics/Learning-in-Ohio/Technology/Ohios-Learning- 
Standards-for-Technology 

Ohio Department of Education. (2019). Ohio's state tests in English language arts, 
mathematics, science, and social studies. 
https://education.ohio.gov/Topics/Testing/Ohios-State-Test-in-ELA-Math-Science- 
SocialStudies 

Ohio Department of Education. (2019). Understanding Ohio’s state test reports 2018-2019. 
https://education.ohio.gov/Topics/Testing/Testing-Results/Results-for-Ohios-State- 
Tests 

Ohio Department of Education. (2020). 2012-2013 and 2013-2014 cut scaled score points for 
basic, proficient, accelerated, and advanced standards, OAA Administration. 
http://education.ohio.gov/Topics/Testing/Testing-Results/Results-for-Ohios-State-Tests 

Ohio Department of Education. (2020). 2014-2015 scale score ranges for all subjects and 
performance levels, PARCC administration. 
http://education.ohio.gov/Topics/Testing/Testing-Results/Results-for-Ohios-State-Tests 

Ohio Department of Education. (2020). Assessment 


committees. https://education.ohio.gov/Topics/Testing/Assessment-Committeescon 


Af is ]_ 


References 


Ohio Department of Education. (2020). Ohio state test name by school year and test 
Subject. http://education.ohio.gov/Topics/Testing/Testing-Results/Results-for-Ohios- 
State-Tests 

Ohio Department of Education. (2020). Statistical summaries and item analysis reports. 
https://education.ohio.gov/Topics/Testing/Statistical-Summaries-and-Item-Analysis- 
Reports 

O'Neill, T., Yamagata, L., Yamagata, J., & Togioka, S. (2012). Teaching STEM means 
teacher learning. The Phi Delta Kappan, 94(1), 36-38. 
https://www.jstor.org/stable/4 1763569 

Pew Research Center. (2018). Women and men in STEM often at odds over workplace 
Equity. https://www.pewsocialtrends.org/2018/01/09/diversity-in-the-stem-workforce- 
varies-widely-across-jobs/ 

Piaget, J. (1971). Psychology and epistemology: Towards a theory of knowledge. Grossman. 

Pinnell, M., Rowley, J., Preiss, S., Blust, R., Beach, R., & Franco, S. (2013). Bridging the 
gap between engineering design and PK-12 curriculum development through the use of 
the STEM education quality framework. Mechanical and Aerospace Engineering 
Faculty Publications, 193(1), 28-35. https://ecommons.udayton.edu/mee_fac_pub/193 

President’s Council of Advisors on Science and Technology. (2010). Prepare and inspire: K- 
12 science, technology, engineering, and math (STEM) education for America’s future. 
https://nsf.gov/attachments/117803/public/2aPrepare_and_ Inspire--PCAST.pdf 

Preuss, M. D., Merriweather, S. P., Walton, S. D. & Butler-Purry, K. L. (2020). Minority 
Student Preparation for STEM PhD Study: Impact of NSF Bridge to the Doctorate 
Programming. Jnternational Journal of Technology in Education and Science (IJTES), 
4(3), 168-187. 

Project Lead the Way. (2020). https://www.pltw.org/our-programs 

Punzalan, C. H. (2022). STEM Interests and Future Career Perspectives of Junior High 
School Students: A Gender Study. Jnternational Journal of Research in Education and 
Science (IJRES), 8(1), 93-102. https://doi.org/10.46328/ijres.2537 

Radloff, J., & Guzey, S. (2016). Investigating preservice STEM teacher conceptions of 
STEM education. Journal of Science Education and Technology, 5(25). 759-774. 
https:www.jstor.org/stable/45151171 

Raudenbush, S., & Bryk, A. (1992). Hierarchical linear models. Sage. 

Raudenbush, S., & Bryk, A. (2002). Hierarchical linear models: Applications and data 
analysis methods (2™ ed.). Sage. 


$$ { 


A Pathway to Success? A Longitudinal Study Using Hierarchical Linear Modeling of Student and 
School Effects on Academic Achievement in a Middle School STEM Program 


Razi, A. & Zhou, G. (2022). STEM, iSTEM, and STEAM: What is next? Jnternational 
Journal of Technology in Education (LUTE), 5(1), 1-29. 
https://doi.org/10.46328/ijte.119 

Sanders, M. (2009). STEM, STEM education, STEMmania. The Technology Teacher, 1(1), 
20-26. 

Saricam, U., & Yildirim, M. (2021). The Effects of Digital Game-based STEM Activities on 
Students’ Interests in STEM Fields and Scientific Creativity: Minecraft Case. 
International Journal of Technology in Education and Science (IJTES), 5(2), 166-192. 
https://doi.org/10.46328/1jtes. 136 

Savery, J. (2006). Overview of problem-based learning: Definitions and distinctions. Journal 
of Problem-based Learning, 1(1). 

Schmid, J. (2007, July). Innovator fears U.S. losing edge; Generac power founder Kern aims 
to revive nation’s engineering education. Milwaukee Journal Sentinel. 

Schwegler, S. (1982). Academic freedom and the disclaimer affidavit of the National 
Defense Education Act: The response of higher education. [Published doctoral 
dissertation]. Teacher’s College, Columbia University. 

Seage, S.J., & Tiiregiin, M. (2020). The Effects of Blended Learning on STEM Achievement 
of Elementary School Students. International Journal of Research in Education and 
Science (IJRES), 6(1), 133-140. 

Seifert, K., & Sutton, R. (2009). Educational psychology (2" ed.). Global Text Project, 33- 
37. https://archive.org/details/EducationalPsychologySecondEdition/page/n1/mode/2up 

Shapiro, M., Grossman, D., Carter, S., Martin, K., Deyton, P., & Hammer, D. (2015). Middle 
school girls and the “leaky pipeline” to leadership. Middle School Journal, 46(5). 3-13. 
https://www.jstor.org/stable/24342229 

Simon, R., Aulls, M., Dedic, H., Hubbard, K., & Hall, N. (2015). Exploring student 
persistence in STEM programs: A motivational model. Canadian Journal of Education, 
38(1), 1-27. https://www.jstor.org/stable/10.2307/canajeducrevucan.38.1.09 

Simsar, A. & Jones, I. (2021). Field Experiences, Mentoring, and Preservice Early Childhood 
Teachers’ Science Teaching Self-Efficacy Beliefs. International Journal on Social and 
Educa-tion Sciences (I[JonSES), 3(3), 518-5534. https://doi.org/10.46328/ijonses.127 

Siregar, N., Rosli, R., Maat, S., and Capraro, M. (2019). The effect of Science, Technology, 


Engineering and Mathematics (STEM) program on students’ achievement in 


AMA ie 


References 


mathematics: A meta-analysis. International Electronic Journal of Mathematics 
Education. 15(1). https://doi.org/10.29333/ieyme/5885 

Snyder, M. (2018). A century of perspectives that influenced the consideration of technology 
as a critical component of STEM education in the United States. Journal of Technology 
Studies, 44(2), 42-57. https://www.jstor.org/stable/10.2307/26730730 

Soap Box Derby. (2020). https://www.soapboxderby.org/ 

Spillane, N. (2015). Teacher characteristics and school-based professional development in 
inclusive STEM-focused high schools [Doctoral dissertation, George Washington 
University]. 

Spillane, N., Lynch, S., & Ford, M. (2016). Inclusive STEM high schools increase 
opportunities for underrepresented students. The Phi Delta Kappan, 97(8), 54-59. 
https://www.jstor.org/stable/24893337 

SPSS. (2020, March 29). In 
Wikipedia. https://en.wikipedia.org/w/index.php?title=SPSS &oldid=94804681 1 

STEM Education Act of 2015, H.R. 1020, 114th Cong. § 59 (2015). 
https://www.congress.gov/bill/1 14th-congress/house-bill/1020/text 

STEM Research and Education Effectiveness and Transparency Act of 2019, S. 2019, 116d 
Cong. §625 (2019). 

Stieff, M., & Uttal, D. (2015). How can spatial training improve STEM achievement? 
Educational Psychology Review, 27(4). https://www.jstor.org/stable/24761210 

Stohlmann, M., Moore, T., & Roehrig, G. (2012). Considerations for teaching integrated 
STEM education. Journal of Pre-College Engineering Education Research, 2(1), 28- 
34. https://doi:10.5703/12882843 14653 

Talan, T. (2021). Augmented Reality in STEM Education: Bibliometric Analysis. 
International Journal of Technology in Education (IJTE), 4(4), 605-623. 
https://doi.org/10.46328/ijte. 136 

Toffler, A. (1970). Future shock. Bantam Books. 

Tschannen-Moran, M., & McMaster, P. (2004). Fostering student learning: The relationship 
of collective teacher efficacy and student achievement. Leadership and Policy in 
Schools, 3(3), 189-209. 

U.S. Census Bureau. (2014, July 10). Census bureau reports majority of STEM college 
graduates do not work in STEM occupations [Press _ release]. 


https://www.census. gov/newsroom/press-releases/2014/cb14-130.html 


SH 


A Pathway to Success? A Longitudinal Study Using Hierarchical Linear Modeling of Student and 
School Effects on Academic Achievement in a Middle School STEM Program 


U.S. Department of Education. (2018). Average NAEP mathematics scores of students in 
Grade 8, by eligibility for national school lunch program: 1996-2017, 2019. Adapted 
from National Center for Science and Engineering Statistics, National Science 
Foundation, special tabulations (2018) of the 1990-2017 NAEP mathematics 
assessments, National Center for Education Statistics, Department of Education. 

U.S. Department of Education. (2018). Average NAEP mathematics scores of students in 
grade 8: 1990-2017, 2019. Adapted from National Center for Science and Engineering 
Statistics, National Science Foundation, special tabulations (2018) of the 1990-2017 
NAEP mathematics assessments, National Center for Education Statistics, Department 
of Education. 

U.S. Department of Education. (2018). Average scores of students in grade 8 on the NAEP 
mathematics assessment, by student characteristics: 2017, 2019. Adapted from 
National Center for Science and Engineering Statistics, National Science Foundation, 
special tabulations (2018) of the 1990-2017 NAEP mathematics assessments, National 
Center for Education Statistics, Department of Education. 

U.S. Department of Education. (2018). Secretary’s final supplemental priorities and 
definitions for discretionary grant programs. Fed. Reg. 83, 42 (March 2, 2018) 
(Notices). https://www.govinfo.gov/content/pkg/FR-2018-03-02/pdf/2018-04291 .pdf 

VentureRadar (2020). STEM education companies. 
https://www.ventureradar.com/keyword/stem%20education 

Wade-Shepard, A. (2016). The effect of middle school STEM curriculum on science and 
math achievement scores. [Doctoral dissertation, Union University]. 

Waight, N., Chisolm, L., & Jacobson, S. (2018). School leadership and STEM enactment in a 
high needs secondary school in Belize. International Studies in Educational 
Administration, 46(1), 102-122. 

Wang, H., Moore, T., Roehrig, G., & Park, M. (2011). STEM integration: Teacher 
perceptions and practice. Journal of Pre-College Engineering Education Research, 
1(2), 1-13. https://doi.org/10.5703/12882843 14636 

Wannapiroon, P., Nilsook, P., Techakosit, S., & Kamkhuntod, S. (2021). STEM Literacy of 
Students in Vocational Education. International Journal of Technology in Education 
and Science (IJTES), 5(4), 527-549. https://doi.org/10.46328/1jtes.253 

White, D. (2014). What is STEM education and why is it important? Florida Association of 
Teacher Educators Journal, 1(14), 1-9. 


A oir 


References 


Wiggins, H. & van der Hoff, Q. (2021). Using an Online Homework System for Fostering 
Self-directed Learning. [International Journal of Technology in Education and Science 
(IJTES), 5(3), 323-335. https://doi.org/10.46328/ijtes.199 

Wilhelm, J. (2014). Project-based instruction with future STEM educators: An 
interdisciplinary approach. Journal of College Science Teaching, 43(4), 80-90. 
https://www.jstor.org/stable/43632018 

Williams, A. M. & Young, J. (2021). Reliability Generalization Meta-analyses in 
Mathematics Education Research: A Research Synthesis. International Journal of 
Education in Mathematics, Science, and Technology (IJEMST), 9(4), 741-759. 
https://doi.org/10.46328/ijemst. 1434 

Woltman, H., Feldstain, A., MacKay, J., & Rocchi, M. (2012). An introduction to 
hierarchical linear modeling. Tutorials in Quantitative Methods for Psychology, 8(1), 
52-69. https://doi: 10.20982/tqmp.08.1.p052 

Yang, D. & Baldwin, S.J. (2020). Using Technology to Support Student Learning in an 
Integrated STEM Learning Environment. Jnternational Journal of Technology in 


Education and Science (IJTES), 4(1), 1-11. 


@ 
CO) 


- A Pathway to Success? A Longitudinal Study Using Hierarchical Linear 
i S te S Modeling of Student and School Effects on Academic Achievement in 


a Middle School STEM Program 
www.istes.org 


APPENDIXES 


Appendix A. Acronyms 


AAAS 


AASBD 


ACC 


AETL 


AIR 


ANCOVA 


API 


DARPA 


ED 


American Association for the Advancement of Science 
All-American Soap Box Derby 

Academic Competitiveness Council 

Advancing Excellence in Technological Literacy 
American Institute of Research 

Analysis of Covariance 

Appalachia Partnership Initiative 

Defense Advanced Research Projects Agency 
Departments of Education 

Engineering by Design 

English Language Arts 

Education Management Information System 
Every Student Succeeds Act 

Falcon Pride Online 

Gravity Racing Challenge 


Health and Human Services 
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Hierarchical Linear Modeling 

Helping Others Pursue Excellence 

Investing in Innovation 

Integrated Postsecondary Education Data System 
International Review Board 

Inclusive STEM High School 

International Technology Education Association 
Innovative Technology Experiences for Students and Teachers 
National Assessment of Educational Progress 
National Aeronautics and Space Administration 
No Child Left Behind 

National Council of Teachers of Mathematics 


National Research Council 


National Science Foundation 

Ohio Department of Education 

Ordinary Least Squares Regression 

Ohio State Tests 

Partnership for Assessment of Readiness for College and Careers 
President’s Council of Advisors on Science and Technology 
Project-Based Learning 

Project Lead the Way 


Science, Math, Engineering, Technology 
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SSAE Student Success and Academic Enrichment Grants 
SSID Student State Identification 

STEM Science, Technology, Engineering, Math 

SLL Standards for Technology Literacy 

TCAP Tennessee Comprehensive Assessment Program 
TfAAp Technology for All Americans Project 


TPACK Technological Pedagogical Content Knowledge 
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Appendix B. Descriptive Statistics for Socioeconomic Status 


Socioeconomic status (SES) disaggregated by STEM participation for 2011-2012 
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(n= 384) 
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(n= 21) 


SES 


Not economically 


disadvantaged 


Economically 
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disadvantaged 


Total 


Frequency 


223 


161 


21 


Percent 


58.1 
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Socioeconomic status (SES) disaggregated by STEM participation for 2012-2013 
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Socioeconomic status (SES) disaggregated by STEM participation for 2013-2014 
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Socioeconomic status (SES) disaggregated by STEM participation for 2014-2015 
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Socioeconomic status (SES) disaggregated by STEM participation for 2015-2016 
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Socioeconomic status (SES) disaggregated by STEM participation for 2016-2017 
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Total 
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420 
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Percent 


59.7 


40.3 
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Socioeconomic status (SES) disaggregated by STEM participation for 2017-2018 
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(n= 49) 
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Socioeconomic status (SES) disaggregated by STEM participation for 2018-2019 
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Appendix C. Descriptive Statistics for Attendance 


Descriptive statistics for student attendance in the 2011-2012 school year 


Mean SD Min. Max. Skew. Kurt. 
Gen. Ed. 9569 0472 .60 1.00 -3.33 18.21 
(n=384 ) 
STEM .9658 0192 94 99 -.18 -1.43 
(n=21 ) 


Descriptive statistics for student attendance in the 2012-2013 school year 


Mean SD Min. Max. Skew. Kurt. 
Gen. Ed. .9610 .0397 65 1.00 -2.60 12.31 
(n= 790) 
STEM .9644 0332 85 1.00 -1.52 2.31 
(n= 52) 


Descriptive statistics for student attendance in the 2013-2014 school year 


Mean SD Min. Max. Skew. Kurt. 
Gen. Ed. 9584 0425 71 1.00 -2.98 15.59 
(n= 1179) 
STEM .9612 .9612 85 1.00 -1.39 2.33 
(n= 74) 


Descriptive statistics for student attendance in the 2014-2015 school year 


Mean SD Min. Max. Skew. Kurt. 
Gen. Ed. 9576 .0480 .60 1.00 -2.64 10.51 
(n= 1075) 
STEM .9675 0447 .70 1.00 -3.72 18.11 
(n=71 ) 
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Descriptive statistics for student attendance in the 2015-2016 school year 


Mean SD Min. Max. Skew. Kurt. 
Gen. Ed. .9615 .0405 63 1.00 -2.38 10.26 
(n=1008 ) 
STEM .9776 0238 89 1.00 -1.53 2.17 
(n= 73) 


Descriptive statistics for student attendance in the 2016-2017 school year 


Mean SD Min. Max. Skew. Kurt. 
Gen. Ed. .9601 0422 64 1.00 -2.06 TAT 
(n=617) 
STEM 9531 .0625 .70 1.00 -2.41 6.03 
(n=52) 


Descriptive statistics for student attendance in the 2017-2018 school year 


Mean SD Min. Max. Skew. Kurt. 
Gen. Ed. .9573 0421 63 1.00 -1.88 7.01 
(n= 554) 
STEM .9600 0580 .70 1.00 .0580 9.87 
(n= 47) 


Descriptive statistics for student attendance in the 2018-2019 school year 


Mean SD Min. Max. Skew. Kurt. 
Gen. Ed. .9527 .0490 44 1.00 -3.91 28.06 
(n= 578) 
STEM 9658 0314 87 1.00 ef 12 1.04 
(n= 45) 


This study discusses the effectiveness of a science, technology, 
engineering, and math (STEM) program spanning grades six through eight 
in a traditional, urban school district located in Northeastern Ohio. The 
history and expressed need for STEM education within post-steel 
producing and economically depressed regions are discussed. Important 
factors describing STEM programming such as curriculum, standards, 
content delivery, integration, and aspects of implementation are 
described. This longitudinal study reports Ohio State Test (OST) scores 
along with the demographic factors of gender, socioeconomic status, 
student race, and attendance rate to determine the impact of STEM 
programming. The use of multilevel, statistical analyses through 
hierarchical linear modeling (HLM) determined integrated STEM had a 
significant, positive effect on student achievement in both math and 
science and an even stronger impact isolating science achievement by 
itself. The predictive results of HLM analysis determined STEM students 
scored significantly higher on the OST in science and math combined 
scoring 31.8 points higher on average and 38.2 points higher in science 
compared to their general education peers. No interaction effects were 
determined between STEM participation and gender, socioeconomic 
status, student race, and attendance rate. This research has powerful 
implications for educational leaders as they need to be aware of the impact 
integrated STEM programming and project-based learning (PBL) has on 

e student achievement. The results indicate that integrated STEM 
programming in middle school has a positive effect on student 
achievement indicating it is indeed a pathway to success. 


ees 9" 781952 " 092220 | 
WV f ©O)/istesoffice ISBN: 978-1-952092-22-0 


